In the challenge Movie review prediction ,the data set size is 40000, after all the stop word removal ,tokenisation ,lemitisation when i go for vectorisation a array of size (40000 x 65k) size is created.
now when i try to fit it in any of the naive bayes algorithm, the following error occurs:
Memory error: unable to allocate 19.6GIB for an array with shape(40000,65742) and datatype float64
even increasing the n_gram range is worthless, it just doubles/triples it’s size
so i want to know how can i solve this problem.
thanks
Pulkit sharma
