Discussion
NDA said:
I do and we're building the analytics team... but we're in London - you're in Yorkshire and the money ain't that big (I don't think) to make the move South.
Can I ask which specific field? I am restricted to Yorkshire but any information would help me target my research into this area.Martin4x4 said:
What makes your think 'Credit Risk' is the right skill set?
Unless you've got very strong enterprise database expertise you are wasting your time.
I wish to try something new but which overlaps somewhat with my skills and experience. I've only recently started reading about the use (or intended use) of big data by lots of companies but there are clear similarities with my current role (data mining with SAS/SQL, time series modelling, regression etc). I don't know what constitutes 'big' data but I often work with moderately large datasets containing a few million rows of data (and hundreds of columns).Unless you've got very strong enterprise database expertise you are wasting your time.
My interest isn't in managing the data infrastructure (which you seem to be alluding to), it's more towards analytics and generating insight.
Having worked in the Business Intelligence arena since before a Data Warehouse was called a Data Warehouse I get a little frustrated with the Vendor managed cycle of renaming the "Same-Old" in order to reinvigorate a product market that is getting a little stale.
BI, in my opinion, has been one of the most consistently exciting areas within business IT for a long time but "size isn't everything". Big Data is just another term vendors have come up with to try and put their products on the top of the performance pile. The reality is, the key players in the different ellements of BI (database, ETL, presentation, analytics) have traded positions consistently over the years without any one really rising above the other in the top 5. They are all istill in business making millions by inventing a new buzzword every 5-10 years.
At the end of the day, it is not about the software (they are all much of a muchness) nor the hardware (sure you need to buy power to proecess and present billions of rows of data) but it is about the analysis and design of that data and the quality of that data in the operation. These two areas are the really hard bits and I challenge you to show me one vendor that says they have that cracked!! Now that is where I specialise and it is discipline where, if followed, the rewards are tremendous. I love my job.....
PS - Working in a Telco where you process and analyse tens of billions of Call Data Records on a daily basis and then conduct extremely detailed fraud analysis is certainly Big Data - But this has been a problem for 30+ years, it wasn't called Big Data back then!!
BI, in my opinion, has been one of the most consistently exciting areas within business IT for a long time but "size isn't everything". Big Data is just another term vendors have come up with to try and put their products on the top of the performance pile. The reality is, the key players in the different ellements of BI (database, ETL, presentation, analytics) have traded positions consistently over the years without any one really rising above the other in the top 5. They are all istill in business making millions by inventing a new buzzword every 5-10 years.
At the end of the day, it is not about the software (they are all much of a muchness) nor the hardware (sure you need to buy power to proecess and present billions of rows of data) but it is about the analysis and design of that data and the quality of that data in the operation. These two areas are the really hard bits and I challenge you to show me one vendor that says they have that cracked!! Now that is where I specialise and it is discipline where, if followed, the rewards are tremendous. I love my job.....
PS - Working in a Telco where you process and analyse tens of billions of Call Data Records on a daily basis and then conduct extremely detailed fraud analysis is certainly Big Data - But this has been a problem for 30+ years, it wasn't called Big Data back then!!
Then of course there is the concept of unstructured data - Sorry, what's the big deal? As the name says, it is obvioulsy in need of structure - You don't need product to do that, you need a bit of common sense and design ability. There are dozens of products that will allow you access but it is again down to analysis, design and quality.....
It's all about scale.
Standard Data > User Derived Data > Big Data
Example:
Standard Data = Basic user input like most office based systems
User Derived Data = Systems where the customers are entering their own data (FaceBook, Twitter, etc).
Big Data = Machines are making data based on other machines and their data. Think Google, the NSA, etc.
TBH, it's a bit of a blur but basically if it's in the million its still considered normal data.
When you get to billions or more you need to use different methods to number crunch and analyse - the technology works across multiple servers, each calculating their own bit and sticking it all back together again. It's only in recent years that systems have been able to grow to such a size so it's become a new buzz-word.
Standard Data > User Derived Data > Big Data
Example:
Standard Data = Basic user input like most office based systems
User Derived Data = Systems where the customers are entering their own data (FaceBook, Twitter, etc).
Big Data = Machines are making data based on other machines and their data. Think Google, the NSA, etc.
TBH, it's a bit of a blur but basically if it's in the million its still considered normal data.
When you get to billions or more you need to use different methods to number crunch and analyse - the technology works across multiple servers, each calculating their own bit and sticking it all back together again. It's only in recent years that systems have been able to grow to such a size so it's become a new buzz-word.
Anubis said:
It's all about scale.
Standard Data > User Derived Data > Big Data
Example:
Standard Data = Basic user input like most office based systems
User Derived Data = Systems where the customers are entering their own data (FaceBook, Twitter, etc).
Big Data = Machines are making data based on other machines and their data. Think Google, the NSA, etc.
TBH, it's a bit of a blur but basically if it's in the million its still considered normal data.
When you get to billions or more you need to use different methods to number crunch and analyse - the technology works across multiple servers, each calculating their own bit and sticking it all back together again. It's only in recent years that systems have been able to grow to such a size so it's become a new buzz-word.
I could go along with this to a certain extent, it is a logical view. However, 18 years ago I ran a programme at a UK fixed-line telco where we processed 120 million rows of data on a daily basis - Held 3 months of detailed history for fraud and tariff analysis purposes and 2 years of aggregated history. The database had 10+billion rows of data that were being routinely interrogated by analysis software.Standard Data > User Derived Data > Big Data
Example:
Standard Data = Basic user input like most office based systems
User Derived Data = Systems where the customers are entering their own data (FaceBook, Twitter, etc).
Big Data = Machines are making data based on other machines and their data. Think Google, the NSA, etc.
TBH, it's a bit of a blur but basically if it's in the million its still considered normal data.
When you get to billions or more you need to use different methods to number crunch and analyse - the technology works across multiple servers, each calculating their own bit and sticking it all back together again. It's only in recent years that systems have been able to grow to such a size so it's become a new buzz-word.
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
NormalWisdom said:
I could go along with this to a certain extent, it is a logical view. However, 18 years ago I ran a programme at a UK fixed-line telco where we processed 120 million rows of data on a daily basis - Held 3 months of detailed history for fraud and tariff analysis purposes and 2 years of aggregated history. The database had 10+billion rows of data that were being routinely interrogated by analysis software.
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
I sympathise with that last comment.
'Data Mining' was a favourite of mine; In other words, your database schema is so poorly designed, and your input validation so lax, that you really don't know what you have and where you put it?
God help us.
I have migrated to Oracle E-Business from so many legacy systems that fall into that category, it makes one wonder how the (business) world manages to keep turning.
That telco you describe sounds much like a system several of my colleagues (being DBAs unlike myself) worked on for quite a few months, about 16-18 years ago, assuming the solution was Oracle based and ran on (at the time) Pyramid I believe?
It was the first time I'd heard the word 'terabyte' used seriously
NormalWisdom said:
I could go along with this to a certain extent, it is a logical view. However, 18 years ago I ran a programme at a UK fixed-line telco where we processed 120 million rows of data on a daily basis - Held 3 months of detailed history for fraud and tariff analysis purposes and 2 years of aggregated history. The database had 10+billion rows of data that were being routinely interrogated by analysis software.
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
Isn't that's just "lots of data" rather than big data?It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
I though big date related to unstructured data rather than big databases
nammynake said:
I wish to try something new but which overlaps somewhat with my skills and experience. I've only recently started reading about the use (or intended use) of big data by lots of companies but there are clear similarities with my current role (data mining with SAS/SQL, time series modelling, regression etc). I don't know what constitutes 'big' data but I often work with moderately large datasets containing a few million rows of data (and hundreds of columns).
My interest isn't in managing the data infrastructure (which you seem to be alluding to), it's more towards analytics and generating insight.
Big Data is by and large a buzz phrase, it is different for different organisations and sectors. Key features is that the data itself is disparate and truly huge. It will likely include structured data from multiple RDBMS, semi-structured documents & forms; new and old (from pdf, digitised documents to scanned microfiche); unstructured e.g letters, video & audio recordings.My interest isn't in managing the data infrastructure (which you seem to be alluding to), it's more towards analytics and generating insight.
Truly huge; terrabytes or petabyes. I worked on a 'big data' system very likely the largest data project in the UK and we never used that term big data. The information was measures in the 100s petabytes of raw data, terrabytes of catalogue data and metadata. (1 petabyte = 1,000 terrabytes = 1,000,000 gigbytes)
The thing you've high lighted here are much more important than the business domain, focus on expanding these analysis and data mining skills. Take a look at sampling theory, map reduction and Hoare logic.
Then if it still appeals take a look at the Digital Library programme at the British Library in Boston Spa.
Edited by Martin4x4 on Wednesday 2nd April 00:05
2222 said:
NormalWisdom said:
I could go along with this to a certain extent, it is a logical view. However, 18 years ago I ran a programme at a UK fixed-line telco where we processed 120 million rows of data on a daily basis - Held 3 months of detailed history for fraud and tariff analysis purposes and 2 years of aggregated history. The database had 10+billion rows of data that were being routinely interrogated by analysis software.
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
Isn't that's just "lots of data" rather than big data?It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
I though big date related to unstructured data rather than big databases
The key here is not the technology it is about getting the design right, doing the analysis and 'cleansing' the sources. I know for a fact that 95% of "Big Data" or "Data Warehouse" projects fail because organisations fail to invest in doing the hard work (Analysis & Design) - And aroung 60% of all BI projects fail to realise their originally stated benefits. This is mainly because the H/W and S/W suppliers have product to sell and cannot impact their margins by getting involved with doing the hard graft. In the last 35 years I have worked for a major H/W vendor selling and delivering Business Intelligence capabilities, I have been a Head of Delivery in an end User responsible for procuring and managing these consulting services and I am now freelance doing a bit of everything. In those 35 years, the problems and solutions have pretty much remained constant wherever I have been ion the world, it's just that they get called different things.
NormalWisdom said:
2222 said:
NormalWisdom said:
I could go along with this to a certain extent, it is a logical view. However, 18 years ago I ran a programme at a UK fixed-line telco where we processed 120 million rows of data on a daily basis - Held 3 months of detailed history for fraud and tariff analysis purposes and 2 years of aggregated history. The database had 10+billion rows of data that were being routinely interrogated by analysis software.
It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
Isn't that's just "lots of data" rather than big data?It's not the concept that irks me it is the creation of a new buzzword to describe what a lot of organisations have done for two decades!
I though big date related to unstructured data rather than big databases
The key here is not the technology it is about getting the design right, doing the analysis and 'cleansing' the sources. I know for a fact that 95% of "Big Data" or "Data Warehouse" projects fail because organisations fail to invest in doing the hard work (Analysis & Design) - And aroung 60% of all BI projects fail to realise their originally stated benefits. This is mainly because the H/W and S/W suppliers have product to sell and cannot impact their margins by getting involved with doing the hard graft. In the last 35 years I have worked for a major H/W vendor selling and delivering Business Intelligence capabilities, I have been a Head of Delivery in an end User responsible for procuring and managing these consulting services and I am now freelance doing a bit of everything. In those 35 years, the problems and solutions have pretty much remained constant wherever I have been ion the world, it's just that they get called different things.
2222 said:
It's not our job to do the hard graft
I do agree but (assuming you work for a vendor in some capacity) it is (IMHO) your job to point out that this graft needs to be done to maximise benfits and that your product alone is not a "silver bullet". Long-term, vendor reputations get damaged but, their products continue to sell like hot-cakes which I just don't get! Actually I do, it's the golf-course, funny handshakes and gentlemens' clubs (says the cynic!)NDA said:
We're involved in R&D in AI, Big Data, machine learning - RTB, have our own DSP, real time profiling... etc. There's moderate action with 12 offices overseas and we are profitable with an EBITDA CAGR over six years of over 1400%.
Who is this then? Can't be many making out there that use machine learning as their big selling point, in fact, Rocketfuel are one of the only ones that spring to my mind that are actively using that as a USP over other DSP/DSN offerings...
ETA no worries, I know who it is now.
okgo said:
Who is this then?
Can't be many making out there that use machine learning as their big selling point, in fact, Rocketfuel are one of the only ones that spring to my mind that are actively using that as a USP over other DSP/DSN offerings...
ETA no worries, I know who it is now.
Can't be many making out there that use machine learning as their big selling point, in fact, Rocketfuel are one of the only ones that spring to my mind that are actively using that as a USP over other DSP/DSN offerings...
ETA no worries, I know who it is now.
Greetings... we should have a conversation at some point Rob.
Gassing Station | Jobs & Employment Matters | Top of Page | What's New | My Stuff