PVD Selenium v4.3 All Scripts

[afrocuban]

As of June 1st, these are the most recent, updated scripts. I have tested them on more than 12.000 titles for several months. Backup your current /Scripts folder and extract this one instead.
Don't forget to download and put appropriate chromedriver.exe into this folder. I couldn't provide it because all of you are using probably different version of Chrome, so you have to do it yourself.

If you don't want to use my way of downloading IMDB Movie additional pages as more reliable and much, much faster one, then you can use jondak's version, which is in addition. I haven't tested it, so I don't know if it works. Just overwright my script in /Scripts folder with jondak's

Extract "StartPVD.7z" in PVD folder, and start PVD with doubleclicking

Code Select Expand

portable.vbs when you don't need logs for debugging, or with doubleclicking

Code Select Expand

debug.vbs when you need log.txt for debugging.

Don't forget to install python, then to put it on PATH, to install selenium, shutil and other modules needed for Python scripts to work properly. Please use Copilot or ChatGPT or any other AI on how to do that, so I could focus on developing and maintaining the scripts. If no AI could help you, post your conversation with AI so I could be sure why AI couldn't help you.

[Ivek23]

Thanks, I'll test it when I can, I'm currently having some minor problems with the electrical wiring. The cause is in my immediate area in a village where there was a lightning strike this afternoon.

[afrocuban]

You are welcome, Ivek. Take your time.

[Ivek23]

IMDB_Movie_[EN][Selenium]-v4 script does not transfer keyword information. I found this error when I did a quick test, the rest of the script works for now, but I will see when I do more tests. The problem is this title below.

10 Things I Hate About You

[afrocuban]

IMDB_Movie_[EN][Selenium]-v4 script does not transfer keyword information. I found this error when I did a quick test, the rest of the script works for now, but I will see when I do more tests. The problem is this title below.

10 Things I Hate About You

I can confirm this. IMDb completely changed html layout for the keywords page.

Here's fixed script for it.

Please let me know if you find out something else.

Best regards

[Ivek23]

I can confirm this. IMDb completely changed html layout for the keywords page.

Here's fixed script for it.

Please let me know if you find out something else.

Okay, I definitely will if I find anything else. I will also publish a slightly revised version in a few days with some code fixes, which therefore also has a better record of the downloaded Imdb data.

[Ivek23]

In Function ParsePage_IMDBMovieBASE the keyword code between lines 3620 and 3833 still doesn't work unless the data from the Plot Keywords web page is loaded.

Otherwise, the data from the Plot Keywords web page is now loaded.

[afrocuban]

Something weird is happening with the Keywords page at the moment. It looks like they're testing new layout. Right now, it's again with "Plot keywords" page title, and this morning it was only "Keywords"

[afrocuban]

Change these two lines accordingly, but I would be patient and wait page layout to stabilize:

Code Select Expand

Line 5967: curPos := Pos('Plot keywords', HTML);
Line 5984: curPos := Pos('Plot keywords', HTML);

[afrocuban]

In Function ParsePage_IMDBMovieBASE the keyword code between lines 3620 and 3833 still doesn't work unless the data from the Plot Keywords web page is loaded.

Can you provide more details which fields aren't working specifically?

[Ivek23]

In Function ParsePage_IMDBMovieBASE the keyword code between lines 3620 and 3833 still doesn't work unless the data from the Plot Keywords web page is loaded.

Can you provide more details which fields aren't working specifically?

My mistake, between lines 3620 and 3683.

This whole code doesn't work when you want to download keywords from the basic Imdb page, not from the page where all the keywords are displayed.

Code Select Expand

        If Not (ReferencePageDownloaded) And Not (KeywordsPageDownloaded) Then Begin
            //Get ~tags~ "keywords" (field with multiple values in a comma separated list)
            curPos := Pos('"keywords":{', HTML);  //WEB_SPECIFIC
            If 0 < curPos Then Begin
                endPos := curPos;
                ItemValue := HTMLValues(HTML, '"keywords":{', '"TitleKeywordConnection"', '"text":"', '","', ', ', endPos);
                AddFieldValueXML('tags', ItemValue);
                LogMessage('Function ParsePage_IMDBMovieBASE - Get results Tags:|' + ItemValue + '||');
            End;  //If 0 < curPos Then Begin should end here
        End;  //If Not (ReferencePageDownloaded) And Not (KeywordsPageDownloaded) Then Begin should end here

        If Not (KeywordsPageDownloaded) Then Begin
            //Extract "Plot Keywords" ~tags~ And (CF~Plot Keywords~) information
            curPos := Pos('data-testid="storyline-plot-keywords"', HTML);
            LogMessage('Function ParsePage_IMDBMovieBASE - Get result curPos: #' + IntToStr(curPos) + '|');
            If 0 < curPos Then Begin
                endPos := PosFrom('data-testid="storyline-plot-links">', HTML, curPos);
                LogMessage('Function ParsePage_IMDBMovieBASE - Get result endPos: #' + IntToStr(endPos) + '|');
            End;  //If 0 < curPos Then Begin should end here

            //Check for "no keywords" message
            If (Pos('It looks like we don' + Chr(39) + 't have any Plot Keywords for this title yet.', HTML) > 0) Then Begin
                LogMessage('Function ParsePage_IMDBMovieBASE - Found no keywords message. Exiting...');
            End;  //If (Pos('It looks like we don' + Chr(39) + 't have any Plot Keywords for this title yet.', HTML) > 0) Then Begin should end here

            //Find the position of the keywords chip list
            curPos := PosFrom('<span class="ipc-chip__text">', HTML, curPos);
            If 0 < curPos Then Begin

                //Initialize counters
                updateCount := 0;
                extraCount := 0;

                //Iterate through all <span class="ipc-chip__text"> tags within the section
                While curPos > 0 Do Begin
                    //Stop if the update count reaches the limit
                    If updateCount >= PLOTKEYWORDS_LIMIT Then Break;

                    //Find the end of the current <span>
                    tagendPos := PosFrom('</span>', HTML, curPos);

                    //Extract the keyword text
                    ItemValue := Copy(HTML, curPos + Length('<span class="ipc-chip__text">'), tagendPos - (curPos + Length('<span class="ipc-chip__text">')));

                    //Check if this is a "... more" link
                    If Pos('more', LowerCase(ItemValue)) > 0 Then Begin
                        tmpStr := Trim(StringReplace(ItemValue, 'more', '', True, True, False));
                        extraCount := StrToIntDef(tmpStr, 0);
                    End Else Begin
                        updateCount := updateCount + 1;
                    End;  //If Pos('more', LowerCase(ItemValue)) > 0 Then Begin should end here

                    //Move curPos past the </span> to find the next one
                    curPos := PosFrom('<span class="ipc-chip__text">', HTML, tagendPos);
                End;  //While curPos > 0 Do Begin should end here

                //Calculate total count including "xxx more"
                totalCount := updateCount + extraCount;

                //Use extracted MoviesID and Category to form the link for (CF~Plot Keywords~)
                AddCustomFieldValueByName('Plot Keywords', '<link url="' + MovieURL + 'keywords/">Plot Keywords</link>' + '      ' + Chr(149) + '     Total Plot keywords / Tags (' + IntToStr(totalCount) + ')' + '     ' + Chr(149));
                LogMessage('Function ParsePage_IMDBMoviePLOTKEYWORDS - Added custom field value for Plot Keywords: <link url="' + MovieURL + 'keywords/">Plot Keywords</link>' + '      ' + Chr(149) + '     Total Plot keywords / Tags (' + IntToStr(totalCount) + ')' + '     ' + Chr(149));
            End;  //If 0 < curPos Then Begin should end here
        End;  //If Not (KeywordsPageDownloaded) Then Begin should end here

[afrocuban]

My mistake, between lines 3620 and 3683.

Oh, ok. I was confused for a moment, hahah. I will check that later, although I checked all html layout from the script from that range and it exists on main page, so I am very curious what wouldn't work then

"keywords":{
"TitleKeywordConnection"
"text":"
data-testid="storyline-plot-keywords"
data-testid="storyline-plot-links">
<span class="ipc-chip__text">
</span>

all still exist on a main page and in proper places...

[Ivek23]

"keywords":{
"TitleKeywordConnection"
"text":"
data-testid="storyline-plot-keywords"
data-testid="storyline-plot-links">
<span class="ipc-chip__text">
</span>

all still exist on a main page and in proper places...

Yes, it's interesting, because I checked it myself, and found everything the same in the source code as stated here. There is something behind the scenes that doesn't let this data be downloaded.

By the way, both your Selenium_Chrome_People_Additional_pages_v4.py script and jondak's Selenium_Chrome_People_Additional_pages_v4.py script work normally.

[afrocuban]

Yes, it's interesting, because I checked it myself, and found everything the same in the source code as stated here. There is something behind the scenes that doesn't let this data be downloaded.

Did you check to Overwrite Tags after restarting PVD (not before!) when choosing Main Page only in Script Configurator?

[Ivek23]

After restarting PVD, I checked the Overwrite Tags option when I selected Main Page Only in the Script Configurator.

However, there was a problem when I selected only the main page in the Script Configurator, the Script Configurator did not save the settings and there was no restart of PVD. It also warned for the PlotSummary page that at least one field associated with this page must be checked. When I checked the fields associated with PlotSummary as well as the PlotSummary additional page, the Script Configurator saved the settings and restarted PVD. Then I checked the Overwrite All Tags option and the keywords from the main page were also transferred.

I also noticed that jondak's Selenium_Chrome_People_Additional_pages_v4.py script may be a little faster at transferring data.

It works with the help of this code in Function ParsePage_IMDBMovieBASE .

//Get ~imdbrating~~IMDB Rating~~IMDBRating~
                ItemValue := TextBetWeenFirst(ItemList,'"worstRating":1,"ratingValue":','},"');  //Strings which opens/closes the data. WEB_SPECIFIC
                //ItemValue := FloatToStr((StrToFloat(ItemValue) * 1));
                AddFieldValueXML('imdbrating',ItemValue);
                AddCustomFieldValueByName('IMDB Rating',ItemValue);
                AddCustomFieldValueByName('IMDBRating',ItemValue);
                LogMessage('Function ParsePage_IMDBMovieBASE - Get result ratingValue ~imdbrating~ (CF~IMDB Rating~~IMDBRating~):|' + ItemValue + '||');
            End;  //If Not (ReferencePageDownloaded) Then Begin should end here
           
            //(*
            If Not (ReferencePageDownloaded) And Not (KeywordsPageDownloaded) Then Begin
                //Get ~tags~ "keywords" (field with multiple values in a comma separated list)
                //curPos := Pos('","keywords":"', HTML);  //WEB_SPECIFIC
                //If 0 < curPos Then Begin
                //    endPos := curPos;
                    //EndPos := PosFrom('","trailer":', HTML, curPos);
                    //ItemValue := HTMLValues(HTML, '"keywords":{', '"TitleKeywordConnection"', '"text":"', '","', ', ', endPos);
                    //ItemValue := TextBetWeen(HTML,'","keywords":"','","trailer":',false,curPos);
                    ItemValue := TextBetWeenFirst(ItemList,'","keywords":"','","');
                    AddFieldValueXML('tags', ItemValue);
                    LogMessage('Function ParsePage_IMDBMovieBASE - Get results Tags:|' + ItemValue + '||');
                //End;  //If 0 < curPos Then Begin should end here
            End;  //If Not (ReferencePageDownloaded) And Not (KeywordsPageDownloaded) Then Begin should end here       
            //*)