Results 1 to 35 of 35

Thread: Big Data Architects

  1. #1
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2

    Big Data Architects

    Probably a long shot as this is a damn senior technology position in a newish skillset... but my team is looking at add more folks for our consulting professional services practice. This is a huge data warehouse vendor which is making a push with Hadoop appliance and services offerings... great customers with lots of different data sets.

    So have you worked extensively with Hadoop? Can you intelligently explain concepts such as RCfile and talk about splittable compression in a map reduce job? Have you worked with HBase, Hive,Oozie, etc? Does standing up a cluster sound like a normal monday? Are you experienced with Java, SQL, python, shell scripting as well? Do you have excellent presentation skills and no problem working directly with customers of all levels?

    If you have, let me know and I will refer you (and likely end up being one of your primary interviewers if you make it past HR).

    This is a 50-100% travel job, can live anywhere and work from home.
    Last edited by CnCGOD; 06-21-2013 at 09:10 PM.

  2. #2
    Registered User
    Join Date
    Dec 2012
    Posts
    83
    Tuconots
    0
    What is the minimum salary for a position like this? $150+?

  3. #3
    Registered User Celestein's Avatar
    Join Date
    Dec 2012
    Posts
    618
    Tuconots
    11
    I can't speak for his company, but as a person who this position would be targeted towards, I probably wouldn't accept less than $180~, and only cause of the work from home aspect, if I was required to move to an expensive city that would change the valuation a lot, clearly.

  4. #4
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Sorry for hijacking the thread but I've always had a few questions about consulting.

    #1) Do you get some form of company matched retirement account?
    #2) Do you get some form of company healthcare?

    I could probably find a consultant job paying decently more per year but my company has an outstanding retirement benefit and health/dental/eyecare programs. It almost feels like I'd need to, at least, double my salary to match all that stuff if consultant firms don't generally off it.

  5. #5
    Registered Dragonlord Deathwing's Avatar
    Join Date
    Dec 2012
    Posts
    6,564
    Tuconots
    31
    Quote Originally Posted by Tenks View Post
    Sorry for hijacking the thread but I've always had a few questions about consulting.

    #1) Do you get some form of company matched retirement account?
    #2) Do you get some form of company healthcare?

    I could probably find a consultant job paying decently more per year but my company has an outstanding retirement benefit and health/dental/eyecare programs. It almost feels like I'd need to, at least, double my salary to match all that stuff if consultant firms don't generally off it.
    I often feel that way too. I got lucky with a job that reimburses everything except $400 of your medical costs per year. A document was sent out recently that valued that benefit at 13% of your salary and I felt that was way too low. Must be a bunch of healthy young people working at my company.

  6. #6
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    Healthcare is pretty standard big company plan which for me alone the high benefit is like 2500$ out of pocket max, 500 deductible for 110$ a month or so.

    401k is I put 6% in and get 4 or 5% matched.

    I know the big consulting companies are similar as I have talked to them myself in the past with varying matches (some even have pensions). These are real high end consulting companies though, not the body shop small ones the field is littered with.
    Last edited by CnCGOD; 06-24-2013 at 07:12 PM.

  7. #7
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Quote Originally Posted by CnCGOD View Post
    Healthcare is pretty standard big company plan. For me alone the high benefit is like 2500$ out of pocket max, 500 deductible for 110$ a month or so.

    401k is e put 6% in and get 4 or 5% matched.
    Ah ok those sound like pretty good benefits then. Was just a question I always had.

  8. #8
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    Ya big vendors and the big pure consulting names (Accenture,Deloitte,etc) all have great benefits. A lot of "consultants" in the industry are really w2 contractors, which is probably what you have seen.

  9. #9
    Everything is meaningless Vinen's Avatar
    Join Date
    Dec 2012
    Posts
    1,288
    Tuconots
    -25
    Man, why would people want to work for such large companies. I'm trying to get out of one right now myself... have a couple of good offers. A paycut is worth not working at a massive company.

  10. #10
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    I think it depends a lot on the company obviously, what is your beef? Bureaucracy?

  11. #11
    Everything is meaningless Vinen's Avatar
    Join Date
    Dec 2012
    Posts
    1,288
    Tuconots
    -25
    Quote Originally Posted by CnCGOD View Post
    I think it depends a lot on the company obviously, what is your beef? Bureaucracy?
    Efficiency overall.

  12. #12
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Yeah the red tape can get pretty annoying at a big company. But not being in charge of every aspect of the operation and not being requested to work 60+ hr/wk is worth it for me.

  13. #13
    Registered User Celestein's Avatar
    Join Date
    Dec 2012
    Posts
    618
    Tuconots
    11
    Quote Originally Posted by Vinen View Post
    Efficiency overall.
    More or less my same issue. For people who take pride in their work, and treat programming like the art it really kind of is, working at the big companies is just soul crushingly depressing, constantly forced to cut corners and compromise your principles.

    It'd be better if the trade-offs were justifiable, but most the time, it's not some well educated PM or CTO making a good call in order to deliver a better product, or to meet a critical deadline. Most the time it's just senseless bureaucracy and meaningless spec-sheets drawn up by some guy trying to justify his existence by producing "something"

    I'm only slightly bitter!

  14. #14
    prescient63
    Guest
    Quote Originally Posted by Tenks View Post
    Yeah the red tape can get pretty annoying at a big company. But not being in charge of every aspect of the operation and not being requested to work 60+ hr/wk is worth it for me.
    Do not go into consulting then. I don't work in tech consulting, but if they work near as hard as we do (strategy consulting) you are going to be putting in some serious hours. I actually wish that I learned to coded because I think that would be much more fun than what I do now. Too late now though!
    Last edited by prescient63; 06-25-2013 at 01:00 PM.

  15. #15
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Quote Originally Posted by Celestein View Post
    More or less my same issue. For people who take pride in their work, and treat programming like the art it really kind of is, working at the big companies is just soul crushingly depressing, constantly forced to cut corners and compromise your principles.

    It'd be better if the trade-offs were justifiable, but most the time, it's not some well educated PM or CTO making a good call in order to deliver a better product, or to meet a critical deadline. Most the time it's just senseless bureaucracy and meaningless spec-sheets drawn up by some guy trying to justify his existence by producing "something"

    I'm only slightly bitter!
    Work for a place where IT is the product they sell. I worked at an insurance company before my current place and it is night and day. IT at the insurance place was seen as this thing that they just were required to have and, if given the chance, would simply cut. Seen as just a black hole of money. Since my current place actually sells what I use they can get real revenue numbers and we're treated much differently. I never have to sacrifice my principals, whatever exactly that means anyways, and am constantly staying on the cutting edge of what the open source community is doing.

  16. #16
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    One of the better aspects of Big Data is the results (analytics) are seen as giving direct business/revenue value... so even the "IT is a cost" places will give it top billing as a money maker.

  17. #17
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    If there is one thing pointy hairs like it is numbers

  18. #18
    Registered Dragonlord Deathwing's Avatar
    Join Date
    Dec 2012
    Posts
    6,564
    Tuconots
    31
    Big businesses usually have better career paths, if you don't like acting like a mercenary and job hopping just to get raises. I'm at a company now that's probably nearing 500 employees and I'm having trouble seeing how I will get promotions to get those sizable raises I know they won't give otherwise.

  19. #19
    Registered User
    Join Date
    Dec 2012
    Posts
    3,713
    Tuconots
    -15
    I've worked at two of the biggest consulting firms around, one in IT, other in mgmt. I'm now independent and will never work for another person again.

    Career path? Hahaha. The biggest joke at the IT firm was that to get a promotion, you had to leave and come back.

  20. #20
    Everything is meaningless Vinen's Avatar
    Join Date
    Dec 2012
    Posts
    1,288
    Tuconots
    -25
    Quote Originally Posted by Dumar View Post
    I've worked at two of the biggest consulting firms around, one in IT, other in mgmt. I'm now independent and will never work for another person again.

    Career path? Hahaha. The biggest joke at the IT firm was that to get a promotion, you had to leave and come back.
    Depends.
    If you are average- you won't get promotions. If you like your work and put an above average effort into it- Promotions AHOY.

    That said-
    Switching jobs is the quickest and most reliable way to get any sort of a monetary raise. I just got promoted and it was only from 120->131. Was hoping for more

  21. #21
    Registered User
    Join Date
    Dec 2012
    Posts
    3,713
    Tuconots
    -15
    You'll get a promotion when you threaten to leave.

  22. #22
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    I've been promoted twice in five years at my company

  23. #23
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    I have found that one of the big things is that most places you have to ask for it to get promoted. Those who don't ask never get promoted.

  24. #24
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Quote Originally Posted by CnCGOD View Post
    I have found that one of the big things is that most places you have to ask for it to get promoted. Those who don't ask never get promoted.
    My first promotion came later than I expected because I didn't speak up. I specifically asked for this next promotion. I believe you are right. I know I wouldn't have been promoted had I not asked.

  25. #25
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    I'm kind of hijacking this thread but CnC do you have any experience with Lucene? Setting it up, indexing your data, using the API? There is a decent sized chance I'm going to join the team for writing our next generation search engine anyways using MapReduce and it appears Lucene is the Apache solution. Are there any other solutions you'd recommend we explore?

  26. #26
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    I have worked with Lucene for keyword indexing of medical documents, it is the goto choice in a lot of deployments. Have you looked into Solr? It has done a lot of the work for you in building the search capabilities on top of the Lucene library.

  27. #27
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    I saw Solr pop up a bunch when I was doing my research into Lucene last night but wasn't 100% sure the function of the product. I haven't joined the team and was just trying to wet my feet to figure out how easy/difficult it is to use MapReduce searches. I'd be on the team investigating solutions so Lucene is just one of the options to explore but it seems to be pretty well regarded as the de facto search for MapReduce.

  28. #28
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    And I'm not sure your knowledge level with Solr but what we really need to support is named indexing. Not just full text keyword searching. In our cause we have data such as:

    Code:
    <Record>
        <001>Spot's adventure</001>
        <002>Dr. Tenks</002>
        <003>Dog</003>
    </Record>
    Bassically the 001, 002 and 003 all have different meanings and conform to the MARC standards set forth by the Library of Congress. So for our "se" index this is for subject which would need to index against the 003. The 001 would be the title index and 002 being the author index. I haven't seen if there is a way of setting something like that up using Solr/Lucene.

  29. #29
    Registered User
    Join Date
    Dec 2012
    Location
    St.Louis, MO
    Posts
    162
    Tuconots
    2
    Quote Originally Posted by Tenks View Post
    And I'm not sure your knowledge level with Solr but what we really need to support is named indexing. Not just full text keyword searching. In our cause we have data such as:

    Code:
    <Record>
        <001>Spot's adventure</001>
        <002>Dr. Tenks</002>
        <003>Dog</003>
    </Record>
    Bassically the 001, 002 and 003 all have different meanings and conform to the MARC standards set forth by the Library of Congress. So for our "se" index this is for subject which would need to index against the 003. The 001 would be the title index and 002 being the author index. I haven't seen if there is a way of setting something like that up using Solr/Lucene.

    I don't have enough lucene experience to tell you if it is good for that. I think you may be looking at an HBase problem with custom indexing.

  30. #30
    Registered User
    Join Date
    Dec 2012
    Posts
    531
    Tuconots
    9
    Quote Originally Posted by Tenks View Post
    And I'm not sure your knowledge level with Solr but what we really need to support is named indexing. Not just full text keyword searching. In our cause we have data such as:

    Code:
    <Record>
        <001>Spot's adventure</001>
        <002>Dr. Tenks</002>
        <003>Dog</003>
    </Record>
    Bassically the 001, 002 and 003 all have different meanings and conform to the MARC standards set forth by the Library of Congress. So for our "se" index this is for subject which would need to index against the 003. The 001 would be the title index and 002 being the author index. I haven't seen if there is a way of setting something like that up using Solr/Lucene.
    I'm not sure if I get what you're writing but Solr indexes are generally segmented against the field where the data came from. So you can use their query language to search against a single field to get your results. There are also cool features you can build against the index like faceting where if you have some date field or word you want to search for you can limit regular search results against them: http://wiki.apache.org/solr/SolrFacetingOverview

    You can see examples of faceting on sites like Newegg where you can limit your search for motherboards to specific subsets of features etc

  31. #31
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    Yeah it isn't really faceting that I'm needing. We have far too many indicies for me to see how we could display facets in a proper UX manner. I am completely ignorant about Solr so I'm just trying to gather some information about the product. I don't even know exactly what a query string looks like you call to it via HTTP or anything. I assume I'd have to at least put a layer in the middle to parse my index names into something usable by Solr.

  32. #32
    Registered User
    Join Date
    Dec 2012
    Posts
    531
    Tuconots
    9
    Quote Originally Posted by Tenks View Post
    Yeah it isn't really faceting that I'm needing. We have far too many indicies for me to see how we could display facets in a proper UX manner. I am completely ignorant about Solr so I'm just trying to gather some information about the product. I don't even know exactly what a query string looks like you call to it via HTTP or anything. I assume I'd have to at least put a layer in the middle to parse my index names into something usable by Solr.
    I'm ignorant of how your data is laid out, but if there is a common identifier for every entry in your source data you could create one singular index for all your data. An example would be a table that has secondary meta data tables that contain extra info for each row. The original row data and all of it's meta data could be put into a single index with all the data segmented into where it was sourced so you can limit your search to the meta data or the main table or both.

    There are some built in query parsers/query syntaxes built into Solr you can choose from (or roll your own), this is the default one and I'm not sure if it helps. http://wiki.apache.org/solr/SolrQuerySyntax

  33. #33
    Registered User
    Join Date
    Dec 2012
    Posts
    12,682
    Tuconots
    54
    I'll try and be a bit more through with the data description then.

    Basically our HBase is keyed off an incremental number as the key. 1, 2, 3 or whatever. I don't know exactly how many keys there are but it is around 100M. Under these keys there are various qualifiers but the big important ones are the XML blob and holdings. Holdings basically means that a library has this book. There are over 1b of these type of entries. The XML blob is a big metadata about the book. So it has things like title, langauges, ISBN/ISSN, author, etc etc. In general here are some common searches:

    "no:123456" --> Returns 123456
    "li:ABC" --> Returns all numbers library ABC has holdings on
    "se:cats" --> Returns all numbers that the /root/040/sa/d contains the phrase "cats" (for this argument lets say the LoC MARC standard for subjects is in the 040 subfield "a")

    There are a bunch of indexes but these are just a few examples. For the most part most of the indexes are similar to the "se:" one where we simply index a field/subfield. I was just wondering if Solr could handle all these use cases.

  34. #34
    Registered User
    Join Date
    Dec 2012
    Posts
    531
    Tuconots
    9
    Yup, and it sounds like it could be a single index. Which from the look of it would greatly simplify your implementation. Take what I say with a grain of salt though, this is all knowledge that I've gained via seeing how Solr would give us a unified search and talking with various companies that do support contracts for Solr. I don't have any first hand experience, but we provide similar searching to what you're describing in our product and the contractors have all given positive feedback that what I described would scale well.

  35. #35
    Registered User
    Join Date
    Dec 2012
    Posts
    270
    Tuconots
    -1
    Re-derailing a bit, but on the consulting topic:

    There is a LOT to be said for being the money maker. If you are an IT or dev at a company where you aren't central to their primary product, you are treated completely differently than if you are the moneymaker. Its pretty easy to justify high pay when you can point at $350k in billable hours in the past year. You also (in my experience) get to do a lot of interesting and cool shit because there are generally 2 scenarios in which people are willing to pay big bucks for a team of consultants:

    1) Shit is totally fucked, and they need to be rescued.
    2) They want to do really cool things but have no fucking clue how to do it and don't trust their own dev team to do it (from what I've seen, this is usually a good choice too).

    fixing #1 can also involve #2, so that scenario isn't all bad.

    The upsides of consulting:
    1) Fast salary growth
    2) do cool shit
    3) work with some really experienced people and able to tap those resources. I was working with a beta SDK from microsoft trying to integrate something into a windows 8 app that wasn't really supposed to be done yet. One of the main resources for that product in the country was an IM away and a guy who worked on silverlight and then windows 8 for the past 5 years was across the room. It would have taken significantly longer without their help.
    4) generally a lot better culture (at our company anyway) than traditional stodgy large companies. Not startup level (although we do have a kegerator) but a long ways from office space. When your business is technology, you tend to get more smart people and fewer pointy haired bosses.

    Downsides:
    1) Can be a lot of travel, depending on the company. Might have to work at the client site even if they are local.
    2) You never "own" a product that you then tinker with for years. You go in, build something (hopefully cool), and then hand it off to the customer.
    3) Not a lot of fuckoff time where you can be lazy and feel like you've "earned it" from previous accomplishments. Those accomplishments were a different team and a different time, very much a "what have you done for me lately" feel to it.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •