Hello everyone! I am diving into the world of Gensim Word2Vec and could use some guidance. My current task involves using Word2Vec to create word vectors for raw HTML files. To kick things off, I convert these HTML files into text files.
Question Number One:
During the training of the Word2Vec model, everything seems to be running smoothly. However, when I attempt to test the accuracy of the model using
model.accuracy(file_name)
An error pops up:
Traceback (most recent call last):
File "build_w2v.py", line 82, in <module>
main()
File "build_w2v.py", line 77, in main
gen_w2v_model()
File "build_w2v.py", line 71, in gen_w2v_model
accuracy = model.accuracy(target)
File "/home/k/shankai/app/anaconda2/lib/python2.7/site-packages/gensim/models/word2vec.py", line 1330, in accuracy
return self.wv.accuracy(questions, restrict_vocab, most_similar, case_insensitive)
File "/home/k/shankai/app/anaconda2/lib/python2.7/site-packages/gensim/models/keyedvectors.py", line 679, in accuracy
raise ValueError("missing section header before line #%i in %s" % (line_no, questions))
ValueError: missing section header before line #0
Here is a snippet from the sample file:
(sample content that needs help with)Upon inspecting the file, it appears to start with multiple unnecessary spaces or newline characters. When viewed in Vim, it presents as shown in this image.
What could possibly be causing this issue?
Second Question:
Additionally, I am working on text classification for biomedical papers which are provided to me in raw HTML format, either in Japanese or English. After converting them to ASCII and cleaning out stop words, I still encounter various remnants of HTML code within the files.
In an attempt to tidy up the files by restricting characters to [a-zA-Z0-9], I noticed that certain medical terms like [4protein...] are not getting cleaned properly.
Any suggestions on how to effectively clean up these files?