This summer I spent my Mitacs Globalink internship at Algoma University building a recommender system that tries to understand something most recommender systems ignore completely: how people actually feel about the things they buy. Most systems know you rated something four stars, or that you bought a moisturizer. What they don’t know is that you loved how it felt on your skin but hated the smell, or that a jacket’s fabric won you over even though the color was wrong.
That gap became the whole project. It’s called EARS, short for Emotion-Aware Recommender System, and over a few months it grew from a rough idea into a full pipeline: an emotion classifier, an aspect-level sentiment extraction stage, a way of building user and item profiles out of both, and a recommender that actually uses all of it. This post is my attempt to walk through what I built, what worked, what didn’t, and why I think the parts that failed were just as useful as the parts that worked.
The problem with star ratings and plain sentiment
Most recommender systems that try to use review text at all tend to stop at binary sentiment. Positive or negative. That throws away almost everything interesting in a review. A single sentence like “the fabric was soft but I hated the color” carries two completely different opinions about two completely different aspects of the same product, and it also carries something closer to actual emotion, not just polarity. Was the person delighted, mildly annoyed, genuinely disgusted?
I wanted to build something that kept both of those layers instead of collapsing them into a single positive or negative score.
Step one: an emotion classifier that actually works on real reviews
The first piece was a classifier that reads a review and assigns it one of five emotions: joy, anger, sadness, disgust, or neutral. This part took longer than I expected, mostly because getting a labeled dataset of real Amazon reviews with trustworthy emotion labels is not something that exists off the shelf.
I ended up building a three-model LLM ensemble (GPT-4o-mini and Gemini-2.5-Flash voting, with Claude Haiku as a tiebreaker) to label a stratified sample of reviews, then had a human manually adjudicate the small number of cases where all three disagreed. I tried a lot of architectures on top of that labeled data, including full fine-tunes of RoBERTa and ModernBERT, before landing on a SetFit-style contrastive approach that turned out to generalize better and make safer mistakes (it almost never flips a positive review into a negative label, which matters a lot more than raw accuracy when the output feeds into a recommender).
Step two: pulling out what people actually liked and disliked
Once I had emotion labels, the next step was aspect-based sentiment extraction. For every review, I used a self-hosted LLM to pull out every distinct opinion: an aspect, an opinion phrase, and a sentiment. Over the full dataset this produced more than a million aspect mentions from about 198,000 reviews, which I then clustered down into a manageable set of around 500 aspect categories.
This is where the “jacket example” comes in, and it’s genuinely the clearest way I have found to explain the whole mechanism. Say a user loved a jacket’s fabric but hated its color. The aspect extraction step catches both of those as separate opinions. When I build that user’s profile, I weight it by how strongly they expressed the emotion (joy counts for a lot more than a mildly positive comment), and then I do something I think is the most important design decision in the whole project: I mask out any aspect the user personally complained about from their own profile. So “color” gets suppressed from their taste profile, but “fabric” stays, because loving the fabric and hating the color are two completely separate signals and treating them as one blended score would wash out both of them.
Negative sentiment enters the system exactly once, right there, purely as personal profile hygiene. It never gets used as a way of matching one user to another. I actually tried six different designs that used negative sentiment as a matching signal, and every single one either did nothing or made recommendations worse. That was a useful thing to learn, not just a dead end.
Step three: the recommender itself
For the actual recommendation engine, I used EASE (a closed-form linear autoencoder) as the collaborative filtering backbone, after comparing it against a full staircase of alternatives (classical CF, neural CF, content-based baselines) and confirming it was the strongest by a clear margin.
Then, for each user, I take the union of EASE’s own top candidates and the aspect and emotion engine’s top candidates, and train a LightGBM model to rerank that combined pool using both collaborative and aspect and emotion features.
That reranker improves NDCG@10 by 25.4 percent over EASE alone, and I didn’t just take that number at face value. I ran a causal ablation to isolate exactly how much of that gain comes from the aspect and emotion signal specifically, rather than just from having two retrieval sources instead of one. The isolated contribution is smaller but real and statistically significant.
The eight ways that didn’t work, and why they mattered
Before I landed on that union-and-rerank design, I tried eight structurally different ways of fusing the aspect and emotion signal directly into the collaborative baseline. Additive score blending, retrieve-then-rerank within an EASE shortlist, reciprocal rank fusion, an unsupervised graph embedding, a relational graph neural network trained end to end, and a jointly-trained two-tower architecture anchored directly to EASE’s own score. All eight failed to beat EASE alone.
I could have just left those out of the story and only shown the design that worked. I didn’t, because the pattern in why they failed turned out to explain the winning design. The jointly-trained architecture in particular taught me something specific: training against uniformly random negatives teaches a model to separate positives from easy negatives, not to refine the ordering among the hard, plausible candidates that actually decide ranking quality at the top. That is exactly why the design that works restricts the reranker to a curated, high-quality candidate pool instead of training end to end over the entire catalog.
A signal-free baseline that ties my primary result
Partway through, I compared everything against SASRec, a sequential recommender that uses chronological interaction order and nothing else. No review text, no ratings, no emotion signal at all. Given how thin most users’ interaction histories are in this dataset, I expected it to lose badly.
It didn’t. Under a fair evaluation protocol, SASRec statistically ties my primary result in aggregate, and it actually wins clearly for users with longer interaction histories, while my system wins clearly for users with thin histories. I tried five different ways of combining the two, and none of them beat the better of the two models alone without a real, disclosed cost (mostly a big hit to novelty and catalog coverage). So I’m reporting it honestly as a complementary finding rather than forcing an integration that would have quietly diluted the actual signal I set out to study.
Explainability and a fairness gap I couldn’t close
Every recommendation the system makes comes with an explanation, grounded directly in the same aspect and emotion data that produced the recommendation, not a generic template. I had this evaluated against a rubric for factual accuracy and structure, and it scored perfectly across a random sample of 150 recommendations.
I also went looking for fairness problems instead of waiting for someone else to find them. There is a real gap: users whose review history leans negative get systematically lower-quality recommendations than users with positive or neutral histories. I tested every fix I could think of within this design (reweighting, masking, widening the candidate pool, blending in popularity) and none of them closed it. My best current read is that this reflects a genuine, structural difference in how learnable those users’ preferences are from this kind of signal, not something a cleverer weighting scheme is going to fix. I think that is worth saying plainly rather than hiding it behind an accuracy-only comparison.
Where things stand
I’ve written this up as a full paper, and as of this post it’s been submitted for review at a conference. Nothing has come back yet, so treat everything here as work in progress rather than a finished, peer-reviewed result. I wanted to write about it now anyway, because the parts I find most worth sharing are not really the headline number. It’s the eight fusion attempts that failed before the design that worked, the SASRec tie I didn’t expect, and the fairness gap I looked for and couldn’t close. Those are the parts that actually taught me something, and I’d rather write about the real process than wait for a stamp of approval to talk about it.