I'm new to the SI data set and MySQL, but familiar with financial data, Excel, VBA, screening, FoxPro (dating way back to dBase II - wow! ), Market Guide data, etc. Was wondering if you have any tips on how to best import SI .dbf tables to MySQL, and how to set things up for good mechanical "grinding" on the data set after that? Here are some specifics I'm struggling with:
1. Having only a background in FoxPro, plus some SQL for DB2 I'm struggling with the numerous and various options for setting up MySQL.
Here is what I'm currently trying out:
machine (developer|server|dedicated Server): server (the performance of 8K records and 5 table joins is still a bit slow)
database (multifunctional|transactional|non-transactional) multifunctional (InnoDB)
InnoDB Tablespace Settings: c: \MySQL\ (had to adjust datadir="C:/MySQL/" in my.ini to make this work. On Visa for the moment - yes, I know, that is indeed unfortunate!)
Connections(20|500|manual): 20
Enable TCP/IP Networking: 3306
Firewall Exception: false
Enable strict mode: true
default character set: (standard|multilingualism|manual): standard (makes Latin1 the default character set)
Install as a service: true
Name: MySQL
Include Bin Directory in Windows PATH: true
Above seems to work O.K., but performance is not all that brisk. (I think OBDC jet in Excel VBA to .dbf tables can do 5 joins on 8k records quicker). However, I may be doing something else wrong that slows things down. Or, MySQL is running fine but the table/query viewer I'm using is slowing down displaying the results. On 8K records with 5 joins (company, sales Q, sales A, mgdesc 2 digit and mdesc 5 digit) on 8k records I'm getting a view in more than a minute.).
2. Thoughts on character encoding?
I'm thinking
default: latin1 -- cp1252 West European
collation sequence? defaults: latin1_sweedish_ci
other option might be to just to with UTF-8 / Unicode all the way around? (or too big?)
3. I managed to mine out the memo fields with ODBC / Jet. So that's O.K. except for some funny characters that seemed to be escaped. Have you dealt with the funny characters?
4. Should each table have a primary key? I speculate yes for performance?
5. The famous hidden column in FoxPro: "_NulFlags". This is an odd binary field that codes if each field in a Fox record contains a null. Do you do anything with this or just ignore it? Are nulls important? Null #0?, Null - not available? Any of this stuff important downstream in computing averages, standard deviations?
For now I just ignored that column but ....?
6. There are many tables with joins on company_ID. Should all these tables be kept separate for some speed reason or would they be better combined into larger tables enabling less joins on the subsequent queries?
7. si_dates are interesting and all over the place. Do you have ways to line up the dates so Q1 is Jan-Feb-Mar, etc.. Otherwise, how do the averages compute on misaligned date data?
8. Do you take the default column data types on the .dbf import? I see that MySQL data types are more numerous. Any glitches here or just take suggested column data type defaults?
9. When you are setting up for backtesting how do you retain the weekly SI data files and organize the MySQL tables into time series files? Can the last updated fields be used? One row per period regardless?
10. In setting up the return calculations do you incorporate dividends, splits, reorgs, etc. I.e. how do you maintain the price & dividend files so you can get total return between any two dates?
Any input on above, or other "data model" tips for MySQL most appreciated!!!
Rick
wow gold
World of Warcraftwow gold,wow gold,wow gold,power leveling wow gold,wow power levelingisan online role-playing experience set in the award-winninguniverse. wow gold,wow gold,WoW Goldadventure,and quest across a vast world. is a "Massively Multiplayer Online Role Playing Game" WoW Gold,wow gold,wow power leveling.wow power levelingwhich allows thousands of players to interact within the same world.Whether adventuring together or fighting against each other in epic battlesplayers will form friendships, wow power leveling,wow power leveling,wow power leveling,wow powerlevelingforge alliances,wow power leveling,wow power levelingand compete with enemies for power leveling and glory.wow gold,wow gold,wow gold,wow gold,wow gold,wow gold,buy wow gold As a wow power leveling online game, enables thousands of players to come together online and battle against theand each other. Cheap WoW Gold,Cheap WoW Gold,cheap wow gold,cheap wow gold,World Of Warcraft gold,Players from across the cheap wow gold,buy wow gold,globe can leave the real world behind and undertake grand quests and heroic exploits in a land of fantastic adventure. world of warcraft gold,world of warcraft gold,world of warcraft gold,Unlike other cheap opponents. Below are some featuresage of conan gold,aoc gold, age of conan gold,aoc gold, age of conan gold,aoc gold,aoc power leveling,cheap aoc gold,buy aoc gold,aoc power level, found in wow powerleveling world of warcraft gold,world of warcraft gold,world of warcraft gold,Unlike other cheap opponents. Below are some featuresaoc gold,age of conan gold,aoc gold, age of conan gold,aoc gold, age of conan gold,aoc gold,aoc power leveling,cheap aoc gold,buy aoc gold,aoc power level, found in wow powerlevelingoil purifier,world of warcraft power leveling,warcraft power leveling,world of warcraft power leveling,warcraft power leveling,ffxi gil,World of warcraft power leveling,ffxi gil,ffxi gil,ffxi gil,ffxi gil,final fantasy xi gil,final fantasy xi gil,final fantasy xi gil,final fantasy xi gil,cheap world of warcraft gold,warcraft gold,cheap world of warcraft gold,warcraft gold,guildwars gold,guildwars gold,guild wars gold,guild wars gold,lotro gold,lotro gold,lotr gold,lotr gold,maplestory mesos,maplestory mesos,maplestory mesos,maplestory mesos, maple story mesos,maple story mesos,maple story mesos,maple story mesos,jewelry store,Maple Story mesos,MapleStory mesos,ms mesos,mesos,SilkRoad Gold,SRO Gold,SilkRoad Online Gold,eq2 plat,eq2 gold,eq2 Platinum,EverQuest 2 Platinum,EverQuest 2 gold,EverQuest 2 plat,lotro gold,lotr gold,Lord of the Rings online Gold,rolex replica,replica rolex,maplestory mesos,maple story mesos,runescape gold,runescape money,rs2 powerleveling,archlord gold,lineage2 adena,Lineage2 powerleveling,lotro powerleveling,chongqing,yantai,aoc gold,age of conan gold,aoc gold, age of conan gold,aoc gold, age of conan gold,aoc gold,aoc power leveling,cheap aoc gold,buy aoc gold,aoc power level,evening dresses,evening gowns,wedding dresses,bridal gowns,wedding gowns,cocktail dresses,Bridesmaid dresses,prom dresses,formal dresses,Chinese Tea,Green Tea,China Tea,Black tea,Oolong Tea,White tea,Herbal Tea,Jasmine tea,Chinese TeaGreen Tea,出国留学,英国留学,留学英国,礼品公司,商务礼品,礼品采购,礼品,网球场,wow cd key,world of warcraft cd key,aoc gold,age of conan gold,promotional products,promotional items,flashlight,环氧地坪,wow gold,wow gold,wow gold,wow gold,wow gold,wow gold,wow gold,wow gold,wow gold,wow power leveling,wow power leveling,wow power leveling,wow power leveling,wow power leveling,wow power leveling,wow gold,world of warcraft power leveling,world of warcraft power leveling,buy aoc gold,cheap aoc gold,aoc gold,age of conan gold,age of conan power leveling,wow gold, wow gold,wow gold, wow gold,wow gold,wow gold, wow gold,wow gold,wow powerleveling,Childrens Clothes,Childrens clothing,Baby Clothing,Baby Clothes,newborn clothes,infant clothes,boys clothes,boys clothing,girls clothing,girls clothes,baby boy clothes,baby boy clothing,baby girl clothes,baby girl clothing,wow power level,wow power level,wow power level