Are your indicators independent? Which measure are you using for independence?

Thank you for your response. That's a great question and the answer may bring up more questions. In this particular chart we are comparing our own index to the global sales of the Semiconductor Industry Association (SIA). What the SIA number represents is a summation of their members' sales reported directly from the companies. We are separately finding data in the supply chain of sales and other measures which approximate total sales (we also have this highly segmented by industry product). So, we are deriving the same value but from different data sources. This implies the two indexes are related -- they are not orthogonal -- but not strictly dependent either. The bigger difference is that our data is available several weeks before the industry numbers are available giving very good predictive value as to expectations for the SIA data. We haven't yet measured the independence because if how we gather the data. To be more clear, if we could find a way to collect exactly what the SIA collects but several weeks earlier, we'd be even happier.

Most of our fundamental data is similar: we are using it to find changes in fundamental data which impacts industries and public companies.

Is there a test you'd recommend we should run?

Apologies for my lack of experience in this field, but surely you've carried out some market research to see who your customers would/could be and who would be interested in your product... cannot you approach those again?

Robert, thanks for your response. Yes I think it'd be great to speak. I sent you a connection note.

Jason, thanks for your thought, and I apologize I wasn't able to answer quickly. Yes, Quandl is an interesting source and I was just offered a senior introduction there. But I think they may be swamped right now as I started speaking to them in December, and then my contact left.

It seems like the cart before the horse. Usually a strategy will need data for backtest. In this case there is data needing a strategy.

Belen, thanks for the response. I did my market research, but for a different group of users -- fundamental analysts. The quant group is new as some had reached out unannounced to me to discuss data: I've never interacted with these clients before (since grad school). And there seems to be a dichotomy of quants with many coming in from the risk side who may not use fundamentals except as a view of the market or economy. I actually just finished a meeting 30 min ago with a senior buy-side risk manager with whom we discussed how to bridge this gap. But I see clearly that true fundamental data -- in a quant-friendly structured data set format, and with high correlations to company/industry data -- should be valuable for quants.

Eric Ross Have you contacted anyone at Dataminr? I'd also suggest getting in touch with Point72 Asset Mgmt (Steve Cohen's family office, they manage his fortune and have been focused combining data science with traditional, fundamental equity analysis for some time now (with algorithms for predictive modeling based on fundamental data where the datasets and resulting algo's show, with statistical significance, the ability to better predict earnings). I can connect you with someone over there via LinkedIn. I'd also touch base with the COO or one of the senior PMs at Renaissance Technologies. Two Sigma is another that's doing this (I have contacts), and I think Bridgewater has been doing it as well. Let's see, many others. Eagle Alpha - they may be interested in buying your data, they sell data and predictive insights to hedge funds. Also I'd reach out to Coatue, Melvin Capital. The list goes on

Why not market it through quandl.com?

Tangentially, you should use a nonlinear, not linear regression, in the displayed scatter plot.

Emma, I sent this response late, late last night but it somehow didn't make it into the conversation yet. I greatly appreciate your thoughtful response, and I've been enjoying seeing your posts in my feed. These are terrific ideas. I know contacts at several of these shops, and there are some I don't know (Eagle Alpha, for instance). But I'd love to communicate with you directly on the others. I'll send you a message. Thanks so much.

John, I appreciate your comment and it's a good thought. (I apologize: I responded late last night but it somehow didn't make it through to the conversation.) I've been thinking of this data in a different way, as a structured way to look at companies and industries based on fundamentals. In a sense, the (simplified) intended strategy is fundamental analysis of companies/stocks. However, I find that when clients get their hands on data -- fundamental data points (such as comments from a CFO), structured fundamental data sets (such as ours, or time-series satellite counts of cars in mall parking lots), or other data -- they can then apply this data to their strategy: the clients/asset managers drive the strategy, and we provide the fundamental data. Indeed our initial conversations with quants entail explaining the data (composition, and potentially how to best use it) and then they go back to their workstations and look how it can fit into their algos/technology/strategy.

Kilian, thanks for your reply. I'm not sure when my replies make through the process and into the conversation, but I had responded to Jason on this earlier and it may not have been available when you responded. Quandl is an interesting source and I was just offered a senior introduction there yesterday. But I think they may be swamped right now -- I started speaking to them in December, and then my contact left abruptly and we had no notice, and they've not really been proactive at subsequently getting to our data. They last said (in April) it would take six weeks before they could even take an initial look at the data we sent in December, even though they know it is for their "Alternative/premium" offerings. (All this and I am still a Quandl user and a fan.)

Steve, I appreciate your insight. I don't see this as tangential as this is a conversation in Quant-focused group. We'll look into this metric/analysis for the data. Thank you for the thought.

Be careful with correlation arguments, in general you will need something more convincing than scatter plots:

http://borkweb.com/story/global-warming-causes-pirate-population-decrease

Julian, thanks for the comment. That's a great chart; I've never seen it before -- the straightest line chart I've ever seen. And a great point: correlation does not equal causation. Going back to how we determine what data to put in our sets: we are not looking at millions of data sets and sorting for the highest correlation. Instead, we are looking for things we can measure/obtain that are direct fundamental drivers of companies/industries. These are things that, if we had a conversation about the data set (as I would as a fundamental Sell-Side analyst) it would be easy to understand why this data was included. We already know (believe for good reason) there is a strong relationship. We simply produce the correlation scatter charts to show that we are at least in the ballpark AND to show that this is structured data a quant can use. Put another way: fundamental investors don't care about correlations; they just want to know what changed and why it matters. Our data helps us do this.

I'm potentially interested...do you have detailed directory of metadata?

Jacob (or Professor Ayres-Thomson), thank you for your interest. We are working on preparing metadata descriptions for publication to clients, but it's not quite ready. In liu of this, I am happy to have a call with you to discuss the data/metadata, its preparation/collection, and implications/impacts. I send you a note.

@Eric, You present a data set to use as model inputs. The independence or correlation if need be can be computed directly from the set. I ask to acertain the level of multicollinearity of the set and the magnitude of information contribution. Succinctly, does each indicator add unique information to the model? Which measure are you utlitizing to answer these questions? Best Wishes

John, thanks for your response and great question. Multicollinearity from what I understand (the majority of my statistics comes from statistical mechanics of chemical/material systems in grad school) is essentially the statistician's bogeyman: it is the correlation of multiple potential independent factors to a third factor (or to each other) and thus potential increase of error. We have a great deal of factors which contribute to these sector-wide indexes, relating to the independent variable (which itself is comprised of a great deal of factors). For instance, using the Electronics Supply Chain chart accompanying this post, there are literally hundreds of inputs into our index, and separately into the independent variable. From a qualitative basis I know fundamentally (from sector expertise) that none of these many factors contribute the majority of the impact of our index (the dependent variable). But we haven't yet calculated this potential impact likely to your satisfaction.

I want to thank everyone who responded to this post. It's been a great conversation, and the insightful replies have been a tremendous help to me. If anyone wants to follow up feel free to contact me. Have a great weekend, and a great holiday weekend (for those of us in the U.S.)