Here's how I got this data.

I started with OpenSubtitles Actually Open Edition. This is a 128GB SQLite database file containing 5,719,123 subtitle files from opensubtitles.org.

The file you're looking for is called opensubs.db and should display as 128GB if you run ll -h.

Now we run an SQL command to get a text file listing all the films available in the database, so we can extract just the English language ones. You can do this by running extract_filetable.sql with a command like cat extract_filetable.txt | sqlite3 > filetable.txt.

The resulting file filetable.txt should be 514,915,237 bytes (514MB) and its md5 should be decb1f1cb48aa931564f951427b0947c. This step took approximately 9 minutes on my computer.

128GB is too unwieldy an amount of data for my purposes, so I decided to keep just the English language files.

You will find that there are a huge number of duplicates. Also, subtitles are tagged as having a certain number of CDs. Based on my investigation, this seems to indicate the number of subtitle files in the archive. Sometimes this is because a movie is split into multiple video files/DVDs, so the subtitle files are for different segments of the same movie. Other times it's for a season of a TV show, and each subtitle file is one episode (and yet other times, individual episodes of TV shows correspond to entirely different rows in the database - there is no consistency there, unfortunately). You also get weird cases like 113482 (The Nightmare Before Christmas) in which the different subtitle files correspond to different languages, even though the archive itself is just tagged as English.

It seems to me that when a 1CD, single subtitle file version is available, we should use that.

Some statistics:

• There are 5,719,123
• There are 889,088 entries in the database tagged as English language
• Of those there are only 301,358 unique entries (distinct title and year)
• Of those, 300,104 have a 1CD subtitle file (and therefore 1,254 that don't), or over 99.5% of all unique entries.

I'm going to proceed with the following assumption: every archive contains exactly one subtitle file.

We run into one issue here. Several of the archive files fail to extract properly, because they contain duplicate files (with the same filename), so the unzip utility wants to know if I should replace the old file or rename the new one. These are the times that happened:

replace unzipped/zips/115589/No.Manners(╟░? [y]es, [n]o, [A]ll, [N]one, [r]ename: 
replace unzipped/zips/215485/(SUBS ENG) Pr+? [y]es, [n]o, [A]ll, [N]one, [r]ename: 
replace unzipped/zips/3308110/Postal The Movie CD 1 ( Germany ).srt? [y]es, [n]o, [A]ll, [N]one, [r]ename: 
replace unzipped/zips/5029053/Dreamcatcher.2003.720p.HDTV.x264.YIFY.srt? [y]es, [n]o, [A]ll, [N]one, [r]ename:
replace unzipped/zips/6097471/Powers.2015.S1E05.720p.WEB-DL.5.1.CRO-DIAMOND.srt? [y]es, [n]o, [A]ll, [N]one, [r]ename: 
replace unzipped/zips/6958962/Savage.Dog.2017.720p.BRRip.850MB.MkvCage_Torrent.UK.srt? [y]es, [n]o, [A]ll, [N]one, [r]ename: 
replace unzipped/zips/85185/Plat? [y]es, [n]o, [A]ll, [N]one, [r]ename:
replace unzipped/zips/8670699/Rammstein - Deutschland (2019).EN.srt? [y]es, [n]o, [A]ll, [N]one, [r]ename:

I will handle these manually.

85185 This is "Platee". It contains two SRTs, which seem to cover two halves of a movie. We will treat this as a 2CD case. 115589 This is "Pumhaeng Zero". It contains two SRTs, two halves of a movie. It is a 2CD set. 215485 2CD set for "Desh Premee"

Note that for the above files, I have to just guess which file is which CD. I'm using whatever one unzip spat out first as CD1, but there's no reason that should be correct. In each case I just rename the files to CD1.srt, CD2.srt.

5029053 This is "Dreamcatcher", and it's a complete 1CD SRT file, just duplicated for no reason. 6097471 This is S1E5 of a show called "Powers". It's a duplicated complete 1CD file. 6958962 "Savage Dog". Duplicated complete 1CD file. 8670699 "Rammstein: Deutschland". Duplicated complete 1CD file.

For the above, I just delete the second, duplicate file that unzip gave me.

3308110 The .nfo claims this is supposed to be a 2CD set for "Murder by Proxy: How America Went Postal", but in fact both files seem to contain basicaly the same text. It seems that we're missing the other half, but what can you do. It's all I have for this movie, so I'll just use it as is, treating it as a 1CD set.

For this, I delete the duplicate file same as above.

I made the above corrections manually.

Other archives could not be unarchived because they contained files with names that were too long:

error:  cannot create unzipped/zips/8257012/╨Ф╨░╤Г. ╨Ш╨╝╨┐╨╡╤А╨╕╤П. ╨Э╨╛╨▓╨╡╨╗╨╗╨░ 1. ╨Т╨╛╨╖╨▓╤А╨░╤Й╨╡╨╜╨╕╨╡ ╨▒╨╗╤Г╨┤╨╜╨╛╨│╨╛ ╤Б╤Л╨╜╨░ [720p, eng subs].srt
error:  cannot create unzipped/zips/9044819/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 1 ╨╕ 2 ╤Б╨╡╤А╨╕╨╕ _ ╨Я╤А╨╡╨╝╤М╨╡╤А╨░ ╤Б.srt
error:  cannot create unzipped/zips/9044820/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 3 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗ .srt
error:  cannot create unzipped/zips/9044828/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 4 ╤Б╨╡╤А╨╕╤П _ C╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4╨║ .srt
error:  cannot create unzipped/zips/9044830/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 6 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗ .srt
error:  cannot create unzipped/zips/9044831/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 7 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4╨║.srt
error:  cannot create unzipped/zips/9044832/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 8 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗ .srt
error:  cannot create unzipped/zips/9044833/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 9 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4╨║.srt
error:  cannot create unzipped/zips/9044834/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 10 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗.srt
error:  cannot create unzipped/zips/9044835/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 11 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4.srt
error:  cannot create unzipped/zips/9044836/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 12 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗.srt
error:  cannot create unzipped/zips/9044837/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 13 ╤Б╨╡╤А╨╕╤П _ C╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4╨║.srt
error:  cannot create unzipped/zips/9044839/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 14 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗.srt
error:  cannot create unzipped/zips/9044841/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 15 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╤Б╨╡╤А╨╕╨░╨╗.srt
error:  cannot create unzipped/zips/9044847/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 16 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4.srt
error:  cannot create unzipped/zips/9044849/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 17 ╤Б╨╡╤А╨╕╤П _ ╨Э╨╛╨▓╤Л╨╣ ╨б╨╡╤А╨╕╨░╨╗.srt
error:  cannot create unzipped/zips/9044851/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 18 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4.srt
error:  cannot create unzipped/zips/9044854/[English] ╨б╨╗╤Г╨│╨░ ╨Э╨░╤А╨╛╨┤╨░ 2 - ╨Ю╤В ╨╗╤О╨▒╨▓╨╕ ╨┤╨╛ ╨╕╨╝╨┐╨╕╤З╨╝╨╡╨╜╤В╨░, 22 ╤Б╨╡╤А╨╕╤П _ ╨б╨╡╤А╨╕╨░╨╗ 2017 ╨▓ 4╨║.srt
error:  cannot create unzipped/zips/9060212/╨Я╨░╤А╤И╨╕╨▓╤Л╨╡ ╨╛╨▓╤Ж╤Л. ╨б╨╡╤А╨╕╤П 2. ╨Т╨╛╨╡╨╜╨╜╤Л╨╣ ╨д╨╕╨╗╤М╨╝. ╨Ф╤А╨░╨╝╨░. ╨Ы╤Г╤З╤И╨╕╨╣ ╨б╨╡╤А╨╕╨░╨╗ ╨Т╤Л╤Е╨╛╨┤╨╜╨╛.srt
error:  cannot create unzipped/zips/9060215/╨Я╨░╤А╤И╨╕╨▓╤Л╨╡ ╨╛╨▓╤Ж╤Л. ╨б╨╡╤А╨╕╤П 4. ╨Т╨╛╨╡╨╜╨╜╤Л╨╣ ╨д╨╕╨╗╤М╨╝. ╨Ф╤А╨░╨╝╨░. ╨Ы╤Г╤З╤И╨╕╨╣ ╨б╨╡╤А╨╕╨░╨╗ ╨Т╤Л╤Е╨╛╨┤╨╜╨╛.srt
error:  cannot create unzipped/zips/9060305/╨Р╨╜╨╜╨░ ╨Я╨░╨▓╨╗╨╛╨▓╨░ 1 ╤Б╨╡╤А╨╕╤П ╨▒╨╕╨╛╨│╤А╨░╤Д╨╕╤З╨╡╤Б╨║╨╕╨╣╨┤╤А╨░╨╝╨░ ╤А╨╡╨╢ ╨н╨╝╨╕╨╗╤М ╨Ы╨╛╤В╤П╨╜╤Г 1983.srt
error:  cannot create unzipped/zips/9060306/╨Р╨╜╨╜╨░ ╨Я╨░╨▓╨╗╨╛╨▓╨░ 2 ╤Б╨╡╤А╨╕╤П ╨▒╨╕╨╛╨│╤А╨░╤Д╨╕╤З╨╡╤Б╨║╨╕╨╣╨┤╤А╨░╨╝╨░ ╤А╨╡╨╢ ╨н╨╝╨╕╨╗╤М ╨Ы╨╛╤В╤П╨╜╤Г 1983.srt

Investigating this second case, we can see that these bad SRT files come from these archives:

dau.the.empire.s01.e01.the.return.of.the.prodigal.son.(2020).eng.1cd.(8257012).zip
servant.of.the.people.s02.e01.episode.2.1.(2017).eng.1cd.(9044819).zip
servant.of.the.people.s02.e03.episode.2.3.(2017).eng.1cd.(9044820).zip
servant.of.the.people.s02.e04.episode.2.4.(2017).eng.1cd.(9044828).zip
servant.of.the.people.s02.e06.episode.2.6.(2017).eng.1cd.(9044830).zip
servant.of.the.people.s02.e07.episode.2.7.(2017).eng.1cd.(9044831).zip
servant.of.the.people.s02.e08.episode.2.8.(2017).eng.1cd.(9044832).zip
servant.of.the.people.s02.e09.episode.2.9.(2017).eng.1cd.(9044833).zip
servant.of.the.people.s02.e10.episode.2.10.(2017).eng.1cd.(9044834).zip
servant.of.the.people.s02.e11.episode.2.11.(2017).eng.1cd.(9044835).zip
servant.of.the.people.s02.e12.episode.2.12.(2017).eng.1cd.(9044836).zip
servant.of.the.people.s02.e13.episode.2.13.(2017).eng.1cd.(9044837).zip
servant.of.the.people.s02.e14.episode.2.14.(2017).eng.1cd.(9044839).zip
servant.of.the.people.s02.e15.episode.2.15.(2017).eng.1cd.(9044841).zip
servant.of.the.people.s02.e16.episode.2.16.(2017).eng.1cd.(9044847).zip
servant.of.the.people.s02.e17.episode.2.17.(2017).eng.1cd.(9044849).zip
servant.of.the.people.s02.e18.episode.2.18.(2017).eng.1cd.(9044851).zip
servant.of.the.people.s02.e22.episode.2.22.(2017).eng.1cd.(9044854).zip
black.sheep.s01.e01.episode.1.1.(2010).eng.1cd.(9060212).zip
black.sheep.s01.e04.episode.1.4.(2010).eng.1cd.(9060215).zip
anna.pavlova.s01.e01.episode.1.1.().eng.1cd.(9060305).zip
anna.pavlova.s01.e02.episode.1.2.().eng.1cd.(9060306).zip

I just used a different tool without the filename limitation to unzip them manually, renamed the offending SRT file to "CD1.srt" in each case (in all cases there was only one SRT file) and then added them to the dataset manually.

We want to know the following:

1. What file formats are used for the subtitle files?
2. What other files (non subtitle files, like .nfo files) are in there?
3. Is it indeed the case, as expected, that every directory contains exactly one subtitle file, except for the ones that are known to be 2CD, 3CD, etc? Does every directory contain the expected number of subtitle files?

I found that none of the archives contain any subdirectories, they're all flat.

I want to make sure the file extensions make sense, so I want to get a list of all file extensions that appear in the entire dataset.

The first thing I did was find all extensions that only appear once in the entire dataset. These are usually files with no extension at all (so some random part of the name, which happens to have dots in it, is interpreted as an extension). If I could parse these with the python srt library, I considered them to be SRT files.

We're going to assume that there are a small, finite number of subtitle file formats. Here are some of the ones I found:

• nfo: These are just plain text files with extra information. Every single archive has one of these.
• srt (288036, after the above correction): SubRip
• ass (10911): A text subtitle format from a program called Aegisub
• txt (1878)
• sub (1674): subtitle format from MicroDVD
• ssa (460): SubStation Alpha
• no extension (no dots in the filename) (287)
• vtt (190): WebVTT, close to but distinct from SRT
• smi (184): SAMI, HTML-like format

We define a movie to be "valid" if it satisfies the following:

• The archive contains exactly one file with the extension .nfo
• All other files are valid subtitle files, i.e. they can be parsed by the Python code
• The number of valid subtitle files equals the number of CDs as determined from the archive's file name

Running check.batch, I found only 151 movie that were deemed invalid by the above criteria.

In about 100 cases, this was due to failing point (2) due to a subtitle fail that could not be parsed without generating warnings from the PySubs2 library. Manual inspection suggested the library can still parse these files without issue, so I ignored these warnings. We'll leave them in the dataset.

In 48 cases, there was an unexpected number of subtitle files (we failed point 3 but passed points 1 and 2). In these cases I just decided to leave it. In many cases this probably indicates that we don't have the complete subtitles for a given movie or TV show, but there's not much we can do about that. Even the movies that pass the above three tests might have this problem. I can't guarantee for example that the subtitle file for a given movie really contains subtitles for that movie, or that they're really in English. This is a level of data correctness that have to just treat as being on a "best effort" basis on my part.

In 20 more cases, there was an error that prevented a subtitle file from being parsed at all (we fail point (2)). I am going to remove these from the dataset, but set them aside in case they can be fixed and added back later.

In many cases - over a thousand - PySubs2 produces an error when parsing a MicroDVD format subtitle file, because it cannot infer the FPS from the file. MicroDVD files give subtitle timings in terms of frames, not seconds. There is a convention for an optional way to specify the FPS in the first line of the file. If this line is not present, PySubs2 complains. We can prevent these errors by essentially telling PySubs2 to not worry about the framerate. However, we would like to know the framerate because we ultimately would like to know the timestamps for each line of dialogue, for the website. Often the framerate can be gotten (hopefully correctly) from the NFO file. So we'll accept these files, but make a note to try to get their FPS from elsewhere later.

I run clean/batch.py to process everything. This applies the above discussed rules to check every subtitle set, and also parses the subtitle files to produce clean text files with just the dialogue, for easier searching.

This takes 90 minutes to 2 hours. I make a note of any errors, and keep:

• All of the original data - the 301338 original directories, with subtitle files and .nfo files.
• All of the processed plain text files
• An SQL database tying everything together - storing in particular the format and encoding of the subtitle files.
• errors.txt, with the list of movies that produced errors (about 30 of them)
• subs.db, an sqlite database
• manifest.txt, basically the database in CSV format