Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Geek Culture / Server Cost - Anyone have information / knowledge about setting up large-scale servers?

Author
Message
RUCCUS
21
Years of Service
User Offline
Joined: 11th Dec 2004
Location: Canada
Posted: 11th Dec 2011 22:21
I am in the process of developing an iPad application under a company Ive recently started with a few investors. Ive come to the conclusion now that this application will eventually need access to a fairly large amount of data, and I would prefer to store this data on a server and have the app retrieve it from the server as it needs it, as opposed to including all of the data into the app and making the app unnecessarily large.

I was wondering if anyone has any experience with servers and any information on good companies to purchase servers from / typical costs of these servers.

As this is an iPad application with the goal of eventually having many users using the app / connecting to the server, Im obviously going to need a fairly powerful system that can handle anywhere from 100 to 500 users a day (Ill eventually have to expand this as the app gains popularity, as I can see this number growing to at least 1000 users a day).

Any help / information is appreciated,
- RUCCUS
BiggAdd
Retired Moderator
21
Years of Service
User Offline
Joined: 6th Aug 2004
Location: != null
Posted: 11th Dec 2011 23:11
Have you looked into Amazon Web Services?
I've been looking into them myself for storing data, and it seems they offer a fairly reasonable price.

MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 11th Dec 2011 23:34
Do you need a server to run an app or simply store ftp files... which become accessed via http port 80?

If so... you just need a file host... aka some web space... unless you mean to get away from using shared space... with a powerful xeon chip running everything... what is the scope of your app? worldwide or just your own country?... Bear in mind that if you are going global... there is huge latency the further users are from your data servers... so I would suggest several data servers for different regions...

Just a thought...

RUCCUS
21
Years of Service
User Offline
Joined: 11th Dec 2004
Location: Canada
Posted: 11th Dec 2011 23:49
The app will initially be launched from and advertised in North America and the states but within the following month or two from launch we're aiming to branch out globally as soon as possible. I dont need the server to run the app, but one of the primary features of the app is the ability for users to select items from a library of many items. This library will be very large, containing somewhere in the order of 3000 files at launch and constantly expanding as more items are added.

Thanks BigAdd, the Amazon SimpleDB service looks interesting, Ill give them a call tomorrow to get some more information. This is all a little new to me, I dont have much experience with databases and servers, Ive never worked on a project big enough to warrant them.

If anyone else has any input Id appreciate it.
bitJericho
23
Years of Service
User Offline
Joined: 9th Oct 2002
Location: United States
Posted: 12th Dec 2011 10:46 Edited at: 12th Dec 2011 10:48
What you're doing sounds actually pretty small-scale. A cheaper VPS would do fine, but it depends on your expected bandwidth.

I'd recommend prgmr.com, which is about half the price of AWS. Slicehost is pretty good, but a bit more expensive too. Gandi.net has the best support you can find, but they are priced about the same as AWS. I'd recommend Gandi over AWS or Slicehost. If you want a service that's more to the metal, so to speak, I'd recommend prgmr. I've been using them for over a year, and they're really stable and fast. If you want a company that does it all, though, I'd go with Gandi.

3000 files is a lot for any web server to transfer though. Can it be compressed into a handful of files? How will you be retrieving these files? Do you need a database or a file server? Does it need to be encrypted? How many simultaneous users do you expect? How much bandwidth do you expect to use? Do you want a bandwidth cap or an open connection?

In some cases, it may be smarter to setup a direct line from an isp and run your server yourself. That would be where you need a dedicated line, a powerful machine, but little redundency. You can also buy a server and then rent space in a datacenter. Either option usually costs about 100usd/mo.


PAGAN_old
20
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 12th Dec 2011 13:20
no experience with any large scale servers, or any servers apart from working wwith win 2003

i recentley took up Free BSD which is used mostly in servers. My friend used FreeBSD for his web server at home and from what i know, BSD is the most Unix-like system out there. Also there are other lunux distros that are also used in large scale servers. For example Novell, Ive been to some IT sceminar several years ago where the novell people pretty much advertised it to everyone, They did talk about it being for large scale servers so i decided to mention it. I cant say anything about any of these since i know very little about servers but these are products that you might try. Also i did read some stuff about storage solutions, Personally i think fibrechannel drives are cool, not sure if thats exactly what you need but fibrechannel technology is for direct networking and exchange of data between the drives over fiberoptic cables. They are the fastest kinds of HDD drives out there The drives themselves are not too expensive but the fibrechannel networking equipment like fiberrouters/ fiberswitches is expensive i (i read it months ago cant remember the exact cost)

the second best thing right now are SAS (serial attached scsi) drives. they are a more recent thing and the second fastest harddrives and slowly catching up to fibrechannel. They seem to be cheaper too. (i was gonna buy myself 2 15k rpm sas drives + sas adapter because i just want to check this stuff out but then i had money trobles).

i dont know if that helped. I think server stuff is cool yet i never had to deal with it.


dont hate people who rip you off,cheat and get away with it, learn from them
bitJericho
23
Years of Service
User Offline
Joined: 9th Oct 2002
Location: United States
Posted: 12th Dec 2011 13:36 Edited at: 12th Dec 2011 13:40
pagan, you're getting into enterprise hardware/software. RUCCUS shouldn't need anything like that unless he's pulling in bank

Nah, really, all he needs is a dinky little server, could be an old desktop machine for all it matters.

What's really at concern here is the infrastructure to choose. I'd personally recommend staying away from AWS. (Cloud computing is a buzz word. Unless you're needing what Amazon offers, you don't actually need it). What Amazon offers is multiple locations, tailored APIs, good redundancy (you have to pay extra though, it's not standard!), one-button scaleability.

If you need an FTP server, aws is overkill


PAGAN_old
20
Years of Service
User Offline
Joined: 28th Jan 2006
Location: Capital of the Evil Empire
Posted: 12th Dec 2011 14:04
Quote: "pagan, you're getting into enterprise hardware/software."


I assumed thats what he meant by large scale servers. But i dont know if "dinky little desktop server" will be enough. if he needs a good server, it needs to be a server.

The cheapest good thing i can think of is a quad CPU AMD Opteron X8 core system. I read some reviews on this a while ago. and even tho they are a lot cheaper and less powerful than intel, everyone seemed to be pleased by it. Also, you want to have at least room for expansion and upgrades so get a motherboard that supports like 32-64 gigs of ram. not sure about this next thing but if you cant get ECC ram, you can try doing a burnin on regular DDR3 ram which kills the less stable bits of memory that cause errors causing them to be ignored so no memory is written to the fried bits. this shouldreduce errors on no ecc ram.

Not sure if supermicro makes AMD boards but they also seem to make good stuff.

Also a good power supply. FSP group makes good server power supplies i own one.

And just as an insurance, its a good idea to get an APC in case power goes out if thats a thing thats important to you.

Btw does anyone know if the latest Intel Xeon came out with a quad-CPU boards yet? last i heard they had only dual cpu boards made by supermicro and the quad CPU was gonna come out soon


dont hate people who rip you off,cheat and get away with it, learn from them
bitJericho
23
Years of Service
User Offline
Joined: 9th Oct 2002
Location: United States
Posted: 12th Dec 2011 14:17 Edited at: 12th Dec 2011 14:27
Well, he said large-scale, but 500 users a day is very small. If he had 500 concurrent users, I would say something like that could need a mid-sized server, depending on the infrastructure needs. There's really not enough info here in the thread to make a good determination.

What I can say, though, is that AWS is probably unnecessary except in extreme circumstances. It's an unnecessarily expensive solution.

For a good recommendation, we'd need to know what kind of files, whether they can be compressed into a handful of files/1 file. How large of files.

An example, could you replace them with HTTP requests that return an xml file, compressed with gzip?

I could get you a server that could do that using nginx, serving 15k simultaneous requests at a time for maybe 100 bucks/mo, depending on a lot of factors. Need redundency? Double or triple that price. Need more? Get your own server and a dedicated line.

Need 1 15mb zipped file to be sent to 10 simultaneous users only when the user opens the program? 10/mo tops, with redundency? 20 bucks. Want it stored in-house? 100 dollar second-hand server and a 100 dollar contract with your cable company, and you can do other stuff with the left-over bandwidth.


RUCCUS
21
Years of Service
User Offline
Joined: 11th Dec 2004
Location: Canada
Posted: 12th Dec 2011 18:02
Bear in mind that for all intensive purposes this app (as any app thats being developed by anyone right now) is aiming to be a top seller in it's category and if the research my company has done is right, it should certainly get there. This is why I dont think a little desktop server being run inside of the office will suffice.

3000 files, each file being at most I would say 10 megs in size. 1000 users / day, each of which retrieving somewhere in the order of 100 of these files / day, meaning... a total transfer of about 100 gigs of data a day. These are obviously just speculations and the numbers will only be getting this high when the app has semi-matured in popularity, so I would preferably want something scalable that I can continue to beef up as the app grows. If things take off the way we hope for, I can see the numbers shooting up to around 50,000 users a day, but thats fairly far down the line and entirely depends on how quickly we begin to advertise globally.

Ill take a look at Gandi tonight, it sounds interesting. Amazon remains to be my choice until then as they have specific information on their website concerning using their services for iphone / ipad apps and they have an API I can use in xCode to connect with their server.
bitJericho
23
Years of Service
User Offline
Joined: 9th Oct 2002
Location: United States
Posted: 12th Dec 2011 18:40
Ah ok, so HTTP is probably out with that amount of volume. SQL isn't designed for 10mb transfers. FTP is probably the way to go. FTP offers good concurrency, scaleability (just make more servers), permissions, all that jazz.

I don't know what the upper level of an FTP server might be, you'd need to research or test. But yeah, FTP sounds like what you need along with no bandwidth cap (find a provider that just limits the speed of your link, like Gandi.)


MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 12th Dec 2011 19:41
If you are looking to test a linux based ftp account on a 50k Mbit network... your welcome to test on my hosting... and see how it works for you to see if ftp will do the trick ... you.can test until you launch... if you want to keep it live on my hosting... we can discuss it at any point... and pretty cheap too... just depends on the total required space hehe...

Login to post a reply

Server time is: 2026-07-27 20:18:22
Your offset time is: 2026-07-27 20:18:22