Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Automatically Grabbing Data

Author
Message
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 20:02
Link
This is Greyhound Racing Data...

http://www.greyhound-data.com/d?r=3400645&z=-5ihvi

It's in Excel format, so it has vertical, and horizontal rows, columns. I want to be able to Auto-grab the data for a race before it starts, and put the Data in DBPro. I can search for spaces to slice the data up, but I was wondering how much work I can get DBPro to do for me.

If it's in Excel format, can I rebuild the Excel page from scratch?

Can DBPro find the site, and the next race?

The data is then used to turn computer dogs into the dogs in the race, and to see if I can simulate the race before it happens.

I am not selling the data, I am using it the way it is meant to be used, to determine the outcome of a race. I don't think I am breaking any laws by using it.

So how much can I automate? Think of that Facebook movie, Zak grabbed all of the data from sites automatically.

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 20:21
Link
I am unable to readily see the data because you need to login, however excel is excel.
I am not sure how often the data is updated in the spreadsheet but if it is not too often you can export data csv format, manually copy into your DB project and use Read / Data to access it.

I know this doesn't fully satisfy your query but hope it helps a little.

Back to top
Profile PM Email Website
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 20:31 Edited at: 3rd Jul 2013 20:32
Link
Oh sorry, didn't know you had to log in. Here is the data on another site. Click on the name of a dog...

http://www.racingpost.com/greyhounds/card.sd#resultDay=2013-07-03&meetingId=18

It's still excel format.

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 20:54
Link
Phew, quite a lot of data.

One method, as before, copy spread sheet data to your desktop excel.
Convert it so that each row begins with "Data" and each subsequent column is separated by a comma. Copy and paste it at the end of your program and use "Read" to get each value when you need.

Alternative method, easier if you don't have to login, is to use the urlmon dll which can copy each web page as text. You can then use IanM's utilities to extract individual info as required.
If you know the url for each webpage that you need then this is useful to call and download each page as required.

Back to top
Profile PM Email Website
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 21:09 Edited at: 3rd Jul 2013 21:15
Link
Sounds good thanks. I will try things out.

This part...
Quote: "One method, as before, copy spread sheet data to your desktop excel"


Can I copy the whole page in one go, or do I have to split it up?

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 21:47
Link
You are able to copy all the info for all dogs for each race, but you would have to do each race 1 at a time.
For any race if you click on "view full race card", you can then copy a lot of info.

Also bad news, the individual dog info is not included in any webpage source code so using the Urlmon dll won't be of much use.

Also please note that copyright rules may also apply.

Back to top
Profile PM Email Website
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 22:01
Link
Quote: "you can then copy a lot of info"


I mean that I don't know how to copy the formatting of Excel so that it loads back into Excel with the rows, and columns in tact.

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 22:28
Link
Click, hold and drag over the info you need, press Ctrl C to copy it, select top left cell in excel and press Ctrl V to paste it. You will need to format the fonts etc but everything should be in their own individual cells.

Repeat for next race and so on.

Back to top
Profile PM Email Website
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 22:31
Link
That's good thanks!

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 22:41 Edited at: 3rd Jul 2013 23:13
Link
Edit - sorry some of this post is wrong, see post below as well.

Then add new column at A, and type the word "Data". Drag the small box in the bottom right down to the last row, this will copy the word "Data" in front of all columns so that DBpro can "Read" it.

Then create a new column just after column B with the first set of data in, select 1st cell of this new blank column and type =B1&"," and press enter.

Drag the small square of cell C1 all the way down to the bottom row and it will add a comma to the end of the data in each cell.
Repeat for other columns of data.

Delete any columns which you don't want or do not have the comma at the end. Select everything, Ctrl A, copy Ctrl C, go to the end of your darkbasic program editor and press Ctrl V to paste it all.

You can now use the DBpro command "Read" to read each row of data.

Back to top
Profile PM Email Website
Battoad
AGK Developer
19
Years of Service
User Offline
Joined: 12th Feb 2007
Location: A Dark Place
Posted: 3rd Jul 2013 23:12
Link
Oops that won't exactly work, although the principle is right.

When you have your initial info in the spread sheet, leave a column or 2 after the last column and then add the word "Data".
Drag the small box down to complete a new column just with the word "Data" in each row.
In the column after "Data" example column J, select J1 and type =B1&","
press enter, this will put the comma at the end of the 1st cell for info showing in B1. drag the small box to duplicate the info in column B but now with the comma at the end.
repeat this for other columns.

this way you will have all your info, comma separated in adjacent columns and suitable for copy and pasting into your DBp editor.

Back to top
Profile PM Email Website
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 3rd Jul 2013 23:13
Link
That's very detailed thanks! There are 24 races to calculate, so I may still need to automate a bit more. Maybe automate the extra columns or something.

Back to top
Profile PM Email Website
Attila
FPSC Reloaded TGC Backer
21
Years of Service
User Offline
Joined: 17th Aug 2004
Location:
Posted: 4th Jul 2013 00:37
Link
Save the data from Excel as CSV, then read it with DB and separate it at the filedseparators.
Back to top
Profile PM
Pincho Paxton
23
Years of Service
User Offline
Joined: 8th Dec 2002
Location:
Posted: 4th Jul 2013 00:44
Link
Ok thanks!

Back to top
Profile PM Email Website
Back to top

Login to post a reply

Server time is: 2026-07-06 18:37:12
Your offset time is: 2026-07-06 18:37:12