Estimate Book Review Entropy from Pre-trained Embeddings

The inspiration for this exploration came from a Twitter (or X) post by Nassim Taleb:

In this article, I propose a simple method to quantify this "summary diversity" or "book dimensionality" using natural language processing (NLP) and a bit match approach. The process involves:

1. Collecting book review data. Since it's impractical to have thousands of books summarized by individuals, I utilize Goodreads reviews as a proxy, trading some accuracy for scalability.
2. Converting each review into numerical embeddings using pre-trained sentence transformers. For each book, this results in a matrix $X$ of dimensions $n \times d$ $n$ is the number of reviews and $d$ is the embedding dimension.
3. Applying Principal Component Analysis (PCA) or equivalently, Singular Value Decomposition (SVD) to the review embeddings: $$ X = USV $$ where $U$ and $V$ are unitary matrices and $S$ is a diagnoal matrix with singular values $s_1 \geq s_2 \geq \cdots \geq s_m$ with $m=\min(n, d)$.
4. Computing the entropy of the singular values: $$ \text{entropy} = -\sum_{i=1}^m p_i \log p_i \text{, where } p_i = \frac{s_i}{\sum_{i=1}^m s_i} $$

The rationale behind this approach is as follows:

• Similar book reviews imply correlated review embeddings $X$, leading to a significant disparity in the singular values of $X$ (i.e. $s_1$ much larger than $s_m$), resulting in low entropy.
• Diversified and independent reviews suggest uncorrelated and orthogonal embeddings $X$ causing the singular values of $X$ to be relatively uniform, leading to high entropy.

This concept mirrors methods used to measure portfolio diversity in finance.

The following figure showcases a comparison of "review entropy" for three thousand popular non-fiction books on Goodreads (y-axis) against their average ratings (x-axis). Intriguingly, there is no correlation between review entropy and ratings. Notably, Nassim Taleb's books exhibit high review entropy yet possess average ratings. I wonder if he have employed a similar analysis and saw these results before sending his tweet lol.

Code

I utilized the Goodreads book review dataset from Kaggle, focusing solely on non-fiction books due to the large data size.

def filter_genre(x):
    return len(x) > 0 and max(x, key=x.get) in ['non-fiction','history, historical fiction, biography']

book_genre = pd.read_json("goodreads_book_genres_initial.json", orient="records", lines=True)
books_filtered = set(book_genre.loc[book_genre['genres'].apply(filter_genre), 'book_id'])

Then, I processed the reviews through pre-trained sentence transformers to generate numerical embeddings. The all-MiniLM-L12-v1 model has an embedding dimension of 384.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L12-v1')
for i, chunk in enumerate(pd.read_json("goodreads_reviews_dedup.json", orient="records", lines=True, chunksize=500000)):
    chunk = chunk.loc[chunk['book_id'].isin(books_filtered), ['book_id', 'rating', 'review_text']]
    chunk = chunk.loc[chunk['review_text'].str.len()>=100]
    if chunk.shape[0] > 0:
      embed = pd.DataFrame(model.encode(chunk['review_text'].tolist(), show_progress_bar=True), index=chunk['book_id'])
      embed.columns = embed.columns.astype(str)
      embed.to_parquet(f"book_review_embeddings/{i}.parquet")

Finally, I computed the SVD for each book, considering those with at least 50 reviews. To ensure fairness, I sampled 50 reviews per book for entropy calculation, as books with larger numbers of reviews (for example 1000) could inherently have higher entropy then the ones with 50-100 reviews.

embed = pd.read_parquet('C:/Users/soars/Downloads/goodreads/book_review_embeddings')
output = pd.DataFrame({'review_count': embed.index.value_counts()})
output = output.loc[output'review_count']>=50]

for i in tqdm(output.index):
    _, s, _ = np.linalg.svd(embed.loc[i].values[:50, :])
    p = s/np.sum(s)
    p = p[p>0]
    output.loc[i, 'entropy'] = -np.dot(p, np.log(p))

Limitations

• Not all reviews are summaries; some discuss editions or translations, especially for classical books which have many different editions. Some reviews, of a biography for example, focus on characters rather than content. For instance, Walter Isaacson's "Steve Jobs" has a moderate review entropy score (~3.63).
• The general-purpose nature of pre-trained embeddings means they also capture reviewers' styles and emotions, which aren't ideal for information entropy measurement. An example is "The Rape of Nanking: The Forgotten Holocaust of World War II" by Iris Chang. This book deals with the Nanjing Massacre, a tragic and brutal episode during the Second Sino-Japanese War where hundreds of thousands of Chinese civilians and disarmed soldiers were murdered, and numerous women were assaulted by soldiers of the Imperial Japanese Army. The intense emotional impact of the subject matter invariably influence the tone of its reviews, leading to a review entropy of only 3.53.
• The methodology's robustness is impacted by the choice of embedding model, dimension, and number of reviews considered. Randomly selecting a subset of reviews also leads to potential information loss. Advanced approaches like random matrix theory could offer better comparisons between books with varying numbers of reviews, but this is outside this article's scope.
• Ultimately, "review diversity" or "book dimensionality" are subjective concepts. This project is a fun exploration rather than a definitive measure of a book's value, as you many already notice in books highlighted in the previous figure --- there is no "skin in the game".

Appendix - books with highest review entropy