Assignment-3 week1 Task done

Assignment-3/Week1/Assignment3_Group7_14_06_2024.md

+# Assignment 3
+
+
+### GitLab: https://git.cs.uni-paderborn.de/cjayesh/DSSE-Group-7/-/tree/main/Assignment-3
+
+
+
+## Week 1 Updates
+
+
+- Assignment 3 Progress:
+
+Tasks Completed: By Abhay Vaghasiya
+
+Downloaded and combined data from provided sources.
+Performed NLP pre-processing: concatenation, text filtering, lemmatization, stemming, and source code removal.
+Calculated the number of terms per post.
+Split dataset into training, validation, and test sets using stratified sampling. Implemented k-fold cross-validation.
+
+Files Generated:
+
+combined_posts.csv
+processed_posts.csv
+train_set.csv
+val_set.csv
+test_set.csv
+train_fold_1.csv to train_fold_5.csv
+val_fold_1.csv to val_fold_5.csv
\ No newline at end of file

Assignment-3/Week1/Average_number_of_terms_per_post.txt

+Average number of terms per post : 0.637909581086686
\ No newline at end of file

Assignment-3/Week1/Programming_posts.rtf

+{\rtf1\ansi\ansicpg1252\cocoartf1347\cocoasubrtf570
+{\fonttbl\f0\fnil\fcharset0 Consolas;}
+{\colortbl;\red255\green255\blue255;\red51\green51\blue51;}
+\paperw11900\paperh16840\margl1440\margr1440\vieww10800\viewh8400\viewkind0
+\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural
+
+\f0\fs26 \cf2 13901960,10841635,10654608,2717596,20872300,3439845,1086607,12428664,1626423,26478962,10139699,4356222,5108399,315736,9208537,19297297,6310153,8586206,6474532,11155176,3831074,20104908,2693394,8175481,6151398,1310520,17014584,11212945,10693955,656934,823148,9293721,2297100,8444267,32020,1330318,5904972,17095134,9925941,1409638,31790,407614,9853453,2042609,11089368,3302714,19385803,4036488,412011,24390433,8013365,5631571,5484326,12296017,13401937,64014,2651852,2406136,5968156,340151,869783,625387,15545607,11039438,16846572,6084009,10317541,5372222,9925953,8363315,11482970,2256806,118931,581118,7453873,1976217,10575430,23934451,643241,6009174,21816562,13500601,7380373,5463503,6798365,11710081,3941349,6073956,19689510,292233,2615316,5031359,8861183,14516191,11126439,3933391,3232281,10018955,2713281,180624,813347,5180217,1394094,5921635,2161674,1052741,4284269,2720703,11459676,57291,13997263,6579756,10341319,12494420,10519075,357095,1282819,3678121,6358745,2587645,3396817,391142,9037716,3500527,2911221,859112,6186090,14515084,3869022,18505197,15597067,4581206,2949817,6187916,399674,122607,2091350,4321990,2629910,1187814,6275265,655234,11342833,4090282,4807171,471115,5108145,899981,13237163,2191489,13833714,7092553,19748129,394298,6133122,6027887,3065135,14266108,6958396,11076790,9162623,465882,6224007,77659,2732840,4704148,1573977,17735831,616658,1399280,3715778,5553146,22094309,20411366,4711361,2067830,9690191,28850042,5904313,5510371,6021612,14699873,7843345,7820225,2046704,3657572,6277027,12458026,2180460,6830581,7209823,108025,4307318,13330067,8182543,391879,900123,7384534,4686451,262925,18407208,11068612,11109163,341462,11088294,1331535,1867426,13082662,426994,900950,3087063,6413003,10709533,12835949,4614121,22551669,408046,234726,646395,18114343,1200346,6554266,14245692,1756541,1425912,287126,2658862,6751331,1253368,1537631,2121194,162028,565062,6106336,5657279,5336279,455687,105499,8443618,6481760,9321451,3466490,6348053,2266461,642137,2792539,7176726,31068,14649745,4767102,1770463,1588471,1820986,3081871,19253471,4484963,7895964,6470492,8219062,75705,657939,301604,19300411,2112188,11984846,11296092,1646801,27312099,6954803,11013802,457668,359331,7593269,6530145,20974811,2412486,2203781,2113461,344590,15607038,9887528,11306929,6742938,3643600,17377593,3857047,4507504,4608367,427954,1242663,2939794,1895732,2046812,6120854,1998824,2951381,615766,1654322,1356051,5338842,8916865,6386117,11568985,4085973,3495903,5848100,1164916,635839,6835358,7907648,5576672,2049035,13224360,7166052,54380,306852,9030071,15575826,10374808,1382088,3214573,3816373,14489423,4899954,13827152,6262494,5465650,3156294,11433913,752337,227557,768095,9399315,2548631,7538988,842415,13257114,1519199,20947836,3632385,3000209,12213665,11131226,824331,546004,3246249,6218838,11979741,298048,7891259,10150312,11272236,1121020,1522853,7486097,6065081,7005482,885883,3522544,6860053,1665976,27411399,5209960,7158067,951204,1517290,29100265,5773011,3440062,399072,650578,3360160,8286775,1768057,21631878,20125277,79482,1826379,13023310,8876745,3224408,19907628,2078291,11512923,12896621,56687,4603787,9588776,5809963,11755605,2675428,5418281,481845,9123457,456109,2842422,989999,10803706,21571598,4440957,3057576,3100410,1990531,720376,1074292,22032912,9832569,3287790,1755206,797960,9095471,530670,2453353,1426179,5200024,8733709,474050,851681,840537,8008672,8150232,6266403,765954,274984,233571,13443022,4047485,10761762,3956467,82099,5606106,6215882,3507253,14542655,612317,3456758,2780956,4213115,4085270,896233,1037201,4544793,29985065,4397249,865240,10493559,2287839,774648,1699751,4635758,3301820,949177,3167932,3416714,2148131,16240747,1158565,272791,7144025,2482270,7231637,9999654,20281144,4578458,15223779,291847,5747987,3187943,12689274,7409158,26127068,421504,2432843,1309767,11102353,877975,7330030,312612,139809,4867457,2479591,8664901,2656233,3726696,3920174,4098771,1573606,9870523,1354749,18418936,2344513,4677876,14699195,12044776,982622,8956193,2374374,21211848,42740,3011378,7007929,1373514,9582733,7781233,12490298,27429099,1478695,2493112,6443238,505981,19354921,9809891,3210620,12390932,4792269,6543123,13572055,27435841,4161896,5866322,7996569,2092112,4738209,22502035,3566476,950214,894369,13951453,6812054,19181284,6852704,2588341,1998882,635838,2842774,8165398,19970734,20264324,14562587,23867,328485,7254796,11584426,18596112,2280247,340521,739268,17284498,3293889,7875889,9889334,7190707,9062475,578970,11360396,6296882,1867285,6530420,2834049,1670187,3623824,12368079,29441068,2671143,4589891,425017,954653,4265602,3732903,447515,8234777,6308668,1031952,1067538,10729348,15088065,10750964,6648244,24148552,950150,485580,3379332,2553690,1178210,2983078,1857659,4982373,480020,6747263,148071,688456,304986,313864,2690863,18750684,2118559,1718934,6296007,7045638,4932722,425978,14602442,594499,5480308,1456084,6587317,11672179,2752431,57701,2260990,16028828,2330381,5100793,16313326,5896849,20565268,2943148,481090,5261515,3213470,5303116,5366229,326339,14401729,12903366,6788240,6143224,566011,3643951,940466,26732515,7232061,1923934,278037,4266079,744474,10273185,4966508,4272879,5814017,229760,23782735,15127262,248162,47683,582251,11021554,2455244,617253,3523833,2253339,4476108,2221546,854395,1006923,5736845,377604,3424895,13976886,18963627,4670369,17141363,1636908,29814919,5685045,12842148,9709141,1039637,10912164,914899,2130053,568916,1167102,10844641,12041246,2213511,3724301,6208998,6680945,218987,4824823,11908153,5263150,2855535,7772378,7897555,6858474,17532052,18271951,3416846,1286864,2565153,1799005,10926621,6193823,2686495,187588,3839783,5074563,279094,5405930,13018295,1369694,1731434,6411372,2280639,7578837,8169114,10749903,400798,381142,5010528,3886988,645268,2551258,970828,5897250,12412163,8155317,12270906,7732371,4291370,2324038,19170846,3534600,9874631,5434975,8048992,14730563,8541485,7530027,10288824,9915658,14864939,262527,1198810,10652097,1420726,4604823,9798806,372140,17146708,6191663,9796187,2810485,6123301,471364,6006759,11716350,5179478,3846107,1646409,3457305,13644584,13596260,1902133,172681,36999,748355,7528455,4626642,211510,9730800,11558237,1968522,2294010,13021536,393509,16125462,17716207,7390286,5786753,612880,7578050,438669,422401,4218691,4181668,157055,1217096,14999088,1006309,7010703,895988,8660357,167852,16103492,265879,1708591,5432163,824912,138298,3346007,5796554,16476771,5468906,1056012,3548602,838117,1096862,1194379,11679651,11916722,2326284,9805228,400115,4728818,3080601,7076576,2292868,9562316,4118654,11881116,21742232,1221121,780880,2158432,396495,7647912,2153490,6384230,2288718,3158657,4223920,15945969,20467904,2727920,13804435,12902110,16335011,7393210,3067735,11010151,5515403,2964436,690177,13036197,2178897,29256154,736845,209795,444010,2870431,26660542,1483178,11008469,703737,11386876,4004279,5336645,325275,8015840,16372888,20365286,626984,6657436,29931525,4050078,23665064,6663888,3101059,13703908,9821626,2776731,46376,4677601,2184806,12278232,1065844,4412618,839577,2073477,4510903,2267744,5203320,7996436,4934563,8217848,1195022,4335577,2504564,7444851,18448,7281272,2507362,18981618,8056517,5658929,570190,2265629,128470,745212,256234,8083979,2063461,3857486,5365570,1044174,4503309,752214,717,3071505,8914731,4511973,116560,7795614,7554984,3700777,11455588,3007383,5455676,1922551,3292908,2179230,1077747,2192772,110451,2188909,7109251,870997,1004717,727030,5001311,5107694,835839,138043,13838777,946114,945620,15234380,13583721,2131796,1095780,13673970,5163902,12242998,2066343,565680,410391,4226701,2450509,13634364,12670395,331615,6697171,5337421,7731146,3100427,4021514,5890438,8361646,2223147,16874549,3120669,1854409,2859111,8317408,16938667,9867265,872794,181029,4963136,3466886,9561504,304713,4194845,941016,30653,3824372,322111,15007281,8184825,19499967,12773339,5808296,2777631,2464478,6259111,18230430,718377,9486835,9129750,7748753,3006117,4272478,7621461,330427,980173,14518059,5559123,4074215,479618,1509536,2624059,1825608,10064630,7827002,3538707,9019128,21875278,2565600,47537,6764311,16025502,3042534,575021,762239,4705005,855282,952712,2378730,7005187,682511,18460016,1038318,13450058,13286416,792160,8846330,20249997,112625,940435,12683546,99242,15926355,2607542,2793383,1247185,13959949,11673361,3698910,1300497,10858907,768492,3277757,330174,1108643,4731987,11074505,128263,1822695,1803065,9264048,6792785,9621592,11669554,2288834,3874512,5552960,13052358,23145040,6340082,13761340,2648550,11140372,2791071,1495541,16382103,5363401,424358,2479277,6178815,4245688,11690191,8685678,6038027,686466,16994238,925525,3937181,3439619,4928601,10374131,2850098,7508502,9593128,4491801,4013779,3956313,1012289,2517825,10399593,3856434,5065281,4545660,2703996,204172,29546537,20776791,5701077,1608706,8270464,4861545,20116573,9599840,3764681,2237755,141781,15098409,3003878,17750319,9266253,1030456,8843579,2987907,2249726,3764212,14613615,12699389,23050120,8515161,19372658,451701,6830137,470111,19292135,10479705,5801128,7099361,742851,298726,4328662,875112,5552695,3083186,8477664,22513658,919287,9599079,12222607,967051,699888,6983854,17502903,65164,11564922,5313027,1081871,2509316,419,1116975,12571052,27203817,4059706,10133454,11099335,9983114,3119596,10670241,204437,2011483,7080447,9764913,172095,7117589,7467632,415216,10424456,11357480,7760712,8412513,5967894,5655673,7767441,7278940,372365,1650857,625725,394829,8574332,498343,6592116,15338199,9675288,1634726,1013428,7474551,1444388,10584526,5482552,3568635,13052793,4659922,2266497,2605140,6263813,364187,30339095,4420894,1163503,3354475,6933833,403868,5624751,8901869,9976560,1869663,21510660,16842713,248221,760820,2636316,15948927,20029083,6494409,12928723,10676517,480504,793042,1756471,1491909,1827059,4046,3458695,2292147,18202999,8224482,15570108,14794393,2289464,5544611,7118592,2123430,2327137,11469211,1326900,10586466,2803209,3711327,16391428,7893535,3460344,1719416,7786086,6649063,20467904,16807889,16582902,6758597,929518,591838,2493557,16567045,8080840,9549582,6649063,20239991,2779757,2869436,1891119,3121597,1226155,5101840,8922532,20514457,9591766,10926903,5287147,10477117,12242671,2516914,4413455,8826994,11112960,992712,10343075,6608086,3488484,1585012,1322878,3611447,530342,502919,6106958,2548726,15828242,9287961,3789359,190066,368494,20091836,2669100,4060684,4195587,14058071,4284621,21325780,4964161,22094706,10332158,4906128,2650652,10110169,4913953,818681,4069596,15688395,6879652,813202,3711327,3066262,6486890,9690766,113860,8441708,10108630,4175428,9095471,4406656,7714211,14195898,30013116,692126,96040,734477,4769574,12698212,4806314,8752731,2593628,4650700,381345,4046350,6509209,3733951,4461254,5865096,3900040,4824799,1019141,8870132,6515018,67167,3119332,279112,4287330,11021551,3577495,6152294,8193518,6634077,4763072,3740193,2430933,14069520,3831702,652134,19717787,1392556,3712017,11295181,7437818,2142813,5714842,19326028,5646419,3074700,14410638,4787181,930124,881132,3621911,1822774,12398027,4253176,3172333,543709,3522248,3140662,16305190,1506858,3237186,8192584,526487,79482,10330998,14798159,5655308,6271112,5178757,612634,11099420,3540210,5103182,368612,881750,748136,3274525,14640045,15581684,14926402,1087428,3491152,2205694,5329750,6890566,262134,4124664,17720079,18342200,9729395,357814,3155953,147669,3309764,12779665,11434530,911882,2210226,2903397,21634523,1102900,2870431,6788240,9809891,14935755,284491,403853,16033834,4718262,9207802,5326005,2959704,2362936,6413366,10502905,280383,8148774,5300389,743675,4774182,4056827,4089507,15402925,6823109,2116870,8745049,5079200,417803,7040838,5634996,2137394,13811179,16284399,2961707,1241331,1652941,14689307,8511933,51589,1927975,10315069,16576754,5350476,323888,1178256,1496497,368001,14928350,11492319,3630329,9440619,819793,11261000,30343269,18460016,2129820,11786265,493883,7878002,11369830,331297,5047196,3726642,10119544,5098672,4863590,1037925,1307182,5399589,6941802,4143884,11322353,14937647,13203458,13390775,4287941,5664331,7840088,5766527,5444006,11542371,13717091,634439,2687366,9612195,4061514,156338,8641799,567316,7713626,11302531,12900023,4408336,94610,1732515,482835,1326886,87542,9028984,776534,9187248,4947781,23228222,7843345,2458631,197482,645208,13532447,1630540,232500,1395897,2649680,3967971,7840283,1345239,1536087,8586720,2013880,1528324,2411184,852860,3956935,37375,546051,12273519,1510185,511793,29810796,734573,18395523,2586159,9642669,12078667,4202964,11167793,6581956,1491123,7197506,5210696,334706,6368502,505237,958879,18418936,6628016,8242837,283145,6852490,6358806,973017,3910364,1192732,2346620,6334323,2025449,5449163,8224482,10260925,1496214,2379917,3058198,5036308,2434906,6386117,1339142,3874422,4256520,12076641,2463532,2159107,325202,3736208,2938435,4413281,7822898,802518,2122798,4210405,5093582,3294483,335719,3840356,2880383,1069,2410221,5544629,600725,2743166,3707749,564541,566570,5487791,2483178,893259,6484,1500141,4753963,4667078,8656089,5773288,140104,895901,4055860,674027,3445219,14557245,6009174,7149074,895786,602345,8315633,3472430,54579,7421612,5164930,561823,1036856,2301216,8366976,942951,698818,1333705,10089682,1971743,967523,2397154,4140920,8383768,4826742,4559991,1791946}
\ No newline at end of file

Assignment-3/Week1/Validation_fold_csv/val_fold_1.csv

+URL ,Purpose,Solution,post_id,category,combined_text,processed_text
+,,,331615,programming, ,
+,,,14058071,programming, ,
+,,,1242663,programming, ,
+,,,12278232,programming, ,
+,,,12242998,programming, ,
+,,,9321451,programming, ,
+,,,11881116,programming, ,
+,,,23934451,programming, ,
+,,,2142813,programming, ,
+,,,4589891,programming, ,
+,,,2118559,programming, ,
+,,,8083979,programming, ,
+,,,11099420,programming, ,
+,,,15338199,programming, ,
+,,,6275265,programming, ,
+,,,13583721,programming, ,
+,,,2607542,programming, ,
+,,,11272236,programming, ,
+,,,869783,programming, ,
+,,,2693394,programming, ,
+,,,3071505,programming, ,
+,,,14489423,programming, ,
+,,,914899,programming, ,
+,,,4491801,programming, ,
+,,,3065135,programming, ,
+,,,10064630,programming, ,
+,,,1081871,programming, ,
+,,,23782735,programming, ,
+,,,989999,programming, ,
+,,,3292908,programming, ,
+,,,4085270,programming, ,
+,,,1187814,programming, ,
+,,,3643951,programming, ,
+,,,6509209,programming, ,
+,,,9030071,programming, ,
+,,,172681,programming, ,
+,,,17502903,programming, ,
+,,,16125462,programming, ,
+,,,4223920,programming, ,
+,,,650578,programming, ,
+,,,3491152,programming, ,
+,,,4982373,programming, ,
+,,,10089682,programming, ,
+,,,18981618,programming, ,
+,,,6187916,programming, ,
+,,,11908153,programming, ,
+,,,950214,programming, ,
+,,,776534,programming, ,
+,,,298726,programming, ,
+,,,11322353,programming, ,
+,,,4021514,programming, ,
+,,,4867457,programming, ,
+,,,3445219,programming, ,
+,,,9095471,programming, ,
+,,,9709141,programming, ,
+,,,9208537,programming, ,
+,,,2551258,programming, ,
+,,,896233,programming, ,
+,,,1770463,programming, ,
+,,,4503309,programming, ,
+,,,6474532,programming, ,
+,,,8286775,programming, ,
+,,,391879,programming, ,
+,,,2406136,programming, ,
+,,,19748129,programming, ,
+,,,6530145,programming, ,
+,,,7530027,programming, ,
+,,,5463503,programming, ,
+,,,1654322,programming, ,
+,,,2265629,programming, ,
+,,,851681,programming, ,
+,,,14649745,programming, ,
+,,,30343269,programming, ,
+,,,10341319,programming, ,
+,,,326339,programming, ,
+,,,7149074,programming, ,
+,,,895988,programming, ,
+,,,2648550,programming, ,
+,,,4321990,programming, ,
+,,,6084009,programming, ,
+,,,2938435,programming, ,
+,,,12773339,programming, ,
+,,,13203458,programming, ,
+,,,19292135,programming, ,
+,,,8641799,programming, ,
+,,,4581206,programming, ,
+,,,950150,programming, ,
+,,,823148,programming, ,
+,,,4711361,programming, ,
+,,,1998824,programming, ,
+,,,6543123,programming, ,
+,,,958879,programming, ,
+,,,3224408,programming, ,
+,,,6823109,programming, ,
+,,,4966508,programming, ,
+,,,2374374,programming, ,
+,,,591838,programming, ,
+,,,2842774,programming, ,
+,,,1198810,programming, ,
+,,,6581956,programming, ,
+,,,1990531,programming, ,
+,,,304986,programming, ,
+,,,4420894,programming, ,
+,,,407614,programming, ,
+,,,4863590,programming, ,
+,,,7486097,programming, ,
+,,,2651852,programming, ,
+,,,1044174,programming, ,
+,,,471115,programming, ,
+,,,3643600,programming, ,
+,,,1608706,programming, ,
+,,,1519199,programming, ,
+,,,9588776,programming, ,
+,,,14613615,programming, ,
+,,,3058198,programming, ,
+,,,8169114,programming, ,
+,,,359331,programming, ,
+,,,4287941,programming, ,
+,,,1902133,programming, ,
+,,,23665064,programming, ,
+,,,6106958,programming, ,
+,,,4161896,programming, ,
+,,,8234777,programming, ,
+,,,3246249,programming, ,
+,,,16372888,programming, ,
+,,,20947836,programming, ,
+,,,21816562,programming, ,
+,,,20104908,programming, ,
+,,,6788240,programming, ,
+,,,18418936,programming, ,
+,,,7045638,programming, ,
+,,,6143224,programming, ,
+,,,5031359,programming, ,
+,,,1517290,programming, ,
+,,,11369830,programming, ,
+,,,7647912,programming, ,
+,,,635838,programming, ,
+,,,688456,programming, ,
+,,,2903397,programming, ,
+,,,2720703,programming, ,
+,,,1708591,programming, ,
+,,,2690863,programming, ,
+,,,646395,programming, ,
+,,,3824372,programming, ,
+,,,703737,programming, ,
+,,,2067830,programming, ,
+,,,403868,programming, ,
+,,,5553146,programming, ,
+,,,14069520,programming, ,
+,,,3042534,programming, ,
+,,,331297,programming, ,
+,,,8664901,programming, ,
+,,,567316,programming, ,
+,,,162028,programming, ,
+,,,2565600,programming, ,
+,,,14542655,programming, ,
+,,,2943148,programming, ,
+,,,14640045,programming, ,
+,,,9874631,programming, ,
+,,,6587317,programming, ,
+,,,4210405,programming, ,
+,,,1392556,programming, ,
+,,,11564922,programming, ,
+,,,2025449,programming, ,
+,,,10709533,programming, ,
+,,,4195587,programming, ,
+,,,899981,programming, ,
+,,,10803706,programming, ,
+,,,14410638,programming, ,
+,,,67167,programming, ,
+,,,10750964,programming, ,
+,,,3789359,programming, ,
+,,,279112,programming, ,
+,,,287126,programming, ,
+,,,7996436,programming, ,
+,,,3568635,programming, ,
+,,,505981,programming, ,
+,,,7878002,programming, ,
+,,,967051,programming, ,
+,,,6758597,programming, ,
+,,,9798806,programming, ,
+,,,2961707,programming, ,
+,,,4213115,programming, ,
+,,,3740193,programming, ,
+,,,2463532,programming, ,
+,,,3287790,programming, ,
+,,,10502905,programming, ,
+,,,21510660,programming, ,
+,,,2669100,programming, ,
+,,,3732903,programming, ,
+,,,46376,programming, ,
+,,,82099,programming, ,
+,,,6263813,programming, ,
+,,,4047485,programming, ,
+,,,2605140,programming, ,
+,,,13703908,programming, ,
+,,,973017,programming, ,

Assignment-3/Week1/Validation_fold_csv/val_fold_2.csv

+URL ,Purpose,Solution,post_id,category,combined_text,processed_text
+,,,4356222,programming, ,
+,,,455687,programming, ,
+,,,19170846,programming, ,
+,,,13082662,programming, ,
+,,,6852490,programming, ,
+,,,1820986,programming, ,
+,,,10133454,programming, ,
+,,,2752431,programming, ,
+,,,8733709,programming, ,
+,,,3121597,programming, ,
+,,,7197506,programming, ,
+,,,14562587,programming, ,
+,,,3100427,programming, ,
+,,,14794393,programming, ,
+,,,8745049,programming, ,
+,,,942951,programming, ,
+,,,8224482,programming, ,
+,,,8224482,programming, ,
+,,,5074563,programming, ,
+,,,1636908,programming, ,
+,,,4036488,programming, ,
+,,,2210226,programming, ,
+,,,2939794,programming, ,
+,,,19907628,programming, ,
+,,,3500527,programming, ,
+,,,1922551,programming, ,
+,,,11755605,programming, ,
+,,,6151398,programming, ,
+,,,17716207,programming, ,
+,,,7176726,programming, ,
+,,,5796554,programming, ,
+,,,4578458,programming, ,
+,,,6224007,programming, ,
+,,,147669,programming, ,
+,,,6792785,programming, ,
+,,,3956935,programming, ,
+,,,3101059,programming, ,
+,,,5449163,programming, ,
+,,,14999088,programming, ,
+,,,10586466,programming, ,
+,,,7380373,programming, ,
+,,,570190,programming, ,
+,,,2553690,programming, ,
+,,,357095,programming, ,
+,,,427954,programming, ,
+,,,2184806,programming, ,
+,,,3856434,programming, ,
+,,,9805228,programming, ,
+,,,855282,programming, ,
+,,,925525,programming, ,
+,,,1509536,programming, ,
+,,,8914731,programming, ,
+,,,20116573,programming, ,
+,,,18230430,programming, ,
+,,,13959949,programming, ,
+,,,14245692,programming, ,
+,,,6368502,programming, ,
+,,,20411366,programming, ,
+,,,17141363,programming, ,
+,,,11433913,programming, ,
+,,,13951453,programming, ,
+,,,11109163,programming, ,
+,,,5444006,programming, ,
+,,,6649063,programming, ,
+,,,5434975,programming, ,
+,,,465882,programming, ,
+,,,824912,programming, ,
+,,,734477,programming, ,
+,,,1069,programming, ,
+,,,10477117,programming, ,
+,,,1634726,programming, ,
+,,,2650652,programming, ,
+,,,457668,programming, ,
+,,,2362936,programming, ,
+,,,4328662,programming, ,
+,,,13052793,programming, ,
+,,,4545660,programming, ,
+,,,4046350,programming, ,
+,,,6634077,programming, ,
+,,,304713,programming, ,
+,,,10479705,programming, ,
+,,,8876745,programming, ,
+,,,6812054,programming, ,
+,,,7278940,programming, ,
+,,,4202964,programming, ,
+,,,1087428,programming, ,
+,,,1163503,programming, ,
+,,,1968522,programming, ,
+,,,9562316,programming, ,
+,,,3119596,programming, ,
+,,,6764311,programming, ,
+,,,5559123,programming, ,
+,,,2803209,programming, ,
+,,,4143884,programming, ,
+,,,8148774,programming, ,
+,,,3416846,programming, ,
+,,,1803065,programming, ,
+,,,5890438,programming, ,
+,,,3522248,programming, ,
+,,,18448,programming, ,
+,,,6006759,programming, ,
+,,,2493557,programming, ,
+,,,15926355,programming, ,
+,,,12835949,programming, ,
+,,,256234,programming, ,
+,,,19326028,programming, ,
+,,,3309764,programming, ,
+,,,1483178,programming, ,
+,,,12698212,programming, ,
+,,,9293721,programming, ,
+,,,12390932,programming, ,
+,,,13804435,programming, ,
+,,,1095780,programming, ,
+,,,1822774,programming, ,
+,,,6941802,programming, ,
+,,,1333705,programming, ,
+,,,2122798,programming, ,
+,,,5065281,programming, ,
+,,,15688395,programming, ,
+,,,748355,programming, ,
+,,,2042609,programming, ,
+,,,3293889,programming, ,
+,,,11690191,programming, ,
+,,,5904313,programming, ,
+,,,1052741,programming, ,
+,,,1121020,programming, ,
+,,,1074292,programming, ,
+,,,682511,programming, ,
+,,,344590,programming, ,
+,,,9729395,programming, ,
+,,,47537,programming, ,
+,,,3440062,programming, ,
+,,,2964436,programming, ,
+,,,10670241,programming, ,
+,,,5487791,programming, ,
+,,,7144025,programming, ,
+,,,992712,programming, ,
+,,,11716350,programming, ,
+,,,11979741,programming, ,
+,,,2869436,programming, ,
+,,,11916722,programming, ,
+,,,4413455,programming, ,
+,,,9821626,programming, ,
+,,,325275,programming, ,
+,,,5646419,programming, ,
+,,,3495903,programming, ,
+,,,7190707,programming, ,
+,,,6954803,programming, ,
+,,,272791,programming, ,
+,,,3816373,programming, ,
+,,,6411372,programming, ,
+,,,11673361,programming, ,
+,,,8586206,programming, ,
+,,,11013802,programming, ,
+,,,7005482,programming, ,
+,,,2450509,programming, ,
+,,,881750,programming, ,
+,,,3354475,programming, ,
+,,,5263150,programming, ,
+,,,10575430,programming, ,
+,,,3006117,programming, ,
+,,,75705,programming, ,
+,,,6358806,programming, ,
+,,,2159107,programming, ,
+,,,13443022,programming, ,
+,,,2430933,programming, ,
+,,,1012289,programming, ,
+,,,9867265,programming, ,
+,,,4686451,programming, ,
+,,,209795,programming, ,
+,,,2148131,programming, ,
+,,,5576672,programming, ,
+,,,4704148,programming, ,
+,,,7875889,programming, ,
+,,,19300411,programming, ,
+,,,2732840,programming, ,
+,,,3460344,programming, ,
+,,,10139699,programming, ,
+,,,875112,programming, ,
+,,,1756471,programming, ,
+,,,6663888,programming, ,
+,,,17014584,programming, ,
+,,,3424895,programming, ,
+,,,5418281,programming, ,
+,,,3764681,programming, ,
+,,,3080601,programming, ,
+,,,5336645,programming, ,
+,,,298048,programming, ,
+,,,1491909,programming, ,
+,,,2180460,programming, ,
+,,,9764913,programming, ,
+,,,3007383,programming, ,
+,,,3534600,programming, ,
+,,,368001,programming, ,
+,,,5103182,programming, ,
+,,,5100793,programming, ,
+,,,8826994,programming, ,

Assignment-3/Week1/Validation_fold_csv/val_fold_3.csv

+URL ,Purpose,Solution,post_id,category,combined_text,processed_text
+,,,1826379,programming, ,
+,,,9440619,programming, ,
+,,,64014,programming, ,
+,,,15581684,programming, ,
+,,,9621592,programming, ,
+,,,5365570,programming, ,
+,,,9870523,programming, ,
+,,,566011,programming, ,
+,,,9832569,programming, ,
+,,,1731434,programming, ,
+,,,1537631,programming, ,
+,,,17284498,programming, ,
+,,,412011,programming, ,
+,,,1019141,programming, ,
+,,,187588,programming, ,
+,,,6334323,programming, ,
+,,,7474551,programming, ,
+,,,2636316,programming, ,
+,,,16313326,programming, ,
+,,,2517825,programming, ,
+,,,11386876,programming, ,
+,,,8056517,programming, ,
+,,,306852,programming, ,
+,,,9287961,programming, ,
+,,,18407208,programming, ,
+,,,7080447,programming, ,
+,,,4667078,programming, ,
+,,,10424456,programming, ,
+,,,9809891,programming, ,
+,,,278037,programming, ,
+,,,5079200,programming, ,
+,,,9264048,programming, ,
+,,,6296007,programming, ,
+,,,8586720,programming, ,
+,,,148071,programming, ,
+,,,1456084,programming, ,
+,,,760820,programming, ,
+,,,951204,programming, ,
+,,,17095134,programming, ,
+,,,7996569,programming, ,
+,,,797960,programming, ,
+,,,23050120,programming, ,
+,,,13036197,programming, ,
+,,,2911221,programming, ,
+,,,14926402,programming, ,
+,,,138298,programming, ,
+,,,1382088,programming, ,
+,,,283145,programming, ,
+,,,1665976,programming, ,
+,,,323888,programming, ,
+,,,4272879,programming, ,
+,,,1356051,programming, ,
+,,,2671143,programming, ,
+,,,21571598,programming, ,
+,,,6271112,programming, ,
+,,,7843345,programming, ,
+,,,4650700,programming, ,
+,,,5108399,programming, ,
+,,,19689510,programming, ,
+,,,10110169,programming, ,
+,,,425017,programming, ,
+,,,2479277,programming, ,
+,,,6266403,programming, ,
+,,,1253368,programming, ,
+,,,881132,programming, ,
+,,,7820225,programming, ,
+,,,634439,programming, ,
+,,,12078667,programming, ,
+,,,108025,programming, ,
+,,,9591766,programming, ,
+,,,5484326,programming, ,
+,,,340521,programming, ,
+,,,8184825,programming, ,
+,,,10761762,programming, ,
+,,,456109,programming, ,
+,,,4559991,programming, ,
+,,,13997263,programming, ,
+,,,3003878,programming, ,
+,,,3874422,programming, ,
+,,,5093582,programming, ,
+,,,12779665,programming, ,
+,,,1420726,programming, ,
+,,,9028984,programming, ,
+,,,7117589,programming, ,
+,,,372365,programming, ,
+,,,5313027,programming, ,
+,,,3956467,programming, ,
+,,,5164930,programming, ,
+,,,8752731,programming, ,
+,,,96040,programming, ,
+,,,5968156,programming, ,
+,,,4964161,programming, ,
+,,,16567045,programming, ,
+,,,139809,programming, ,
+,,,7390286,programming, ,
+,,,5337421,programming, ,
+,,,11131226,programming, ,
+,,,9399315,programming, ,
+,,,10288824,programming, ,
+,,,6384230,programming, ,
+,,,3764212,programming, ,
+,,,1646801,programming, ,
+,,,335719,programming, ,
+,,,481090,programming, ,
+,,,4507504,programming, ,
+,,,7421612,programming, ,
+,,,5801128,programming, ,
+,,,6657436,programming, ,
+,,,1345239,programming, ,
+,,,2432843,programming, ,
+,,,11512923,programming, ,
+,,,262925,programming, ,
+,,,3711327,programming, ,
+,,,2063461,programming, ,
+,,,1670187,programming, ,
+,,,11074505,programming, ,
+,,,6218838,programming, ,
+,,,12458026,programming, ,
+,,,8477664,programming, ,
+,,,594499,programming, ,
+,,,13023310,programming, ,
+,,,2073477,programming, ,
+,,,568916,programming, ,
+,,,4826742,programming, ,
+,,,29810796,programming, ,
+,,,2780956,programming, ,
+,,,211510,programming, ,
+,,,417803,programming, ,
+,,,7827002,programming, ,
+,,,6009174,programming, ,
+,,,4824823,programming, ,
+,,,20239991,programming, ,
+,,,17146708,programming, ,
+,,,12494420,programming, ,
+,,,5624751,programming, ,
+,,,5399589,programming, ,
+,,,1226155,programming, ,
+,,,2116870,programming, ,
+,,,396495,programming, ,
+,,,8219062,programming, ,
+,,,842415,programming, ,
+,,,3657572,programming, ,
+,,,894369,programming, ,
+,,,5098672,programming, ,
+,,,2046812,programming, ,
+,,,5658929,programming, ,
+,,,3698910,programming, ,
+,,,2324038,programming, ,
+,,,2410221,programming, ,
+,,,6890566,programming, ,
+,,,1822695,programming, ,
+,,,13286416,programming, ,
+,,,12571052,programming, ,
+,,,56687,programming, ,
+,,,9925953,programming, ,
+,,,8270464,programming, ,
+,,,2397154,programming, ,
+,,,9266253,programming, ,
+,,,18418936,programming, ,
+,,,498343,programming, ,
+,,,530670,programming, ,
+,,,1506858,programming, ,
+,,,4074215,programming, ,
+,,,3724301,programming, ,
+,,,26478962,programming, ,
+,,,5657279,programming, ,
+,,,9889334,programming, ,
+,,,3632385,programming, ,
+,,,4284621,programming, ,
+,,,748136,programming, ,
+,,,1096862,programming, ,
+,,,2237755,programming, ,
+,,,19372658,programming, ,
+,,,128470,programming, ,
+,,,2516914,programming, ,
+,,,10018955,programming, ,
+,,,6215882,programming, ,
+,,,9019128,programming, ,
+,,,13018295,programming, ,
+,,,2153490,programming, ,
+,,,9999654,programming, ,
+,,,2951381,programming, ,
+,,,3548602,programming, ,
+,,,7554984,programming, ,
+,,,1425912,programming, ,
+,,,911882,programming, ,
+,,,752214,programming, ,
+,,,2346620,programming, ,
+,,,8656089,programming, ,
+,,,399674,programming, ,
+,,,967523,programming, ,
+,,,4774182,programming, ,
+,,,3967971,programming, ,
+,,,20974811,programming, ,
+,,,13237163,programming, ,
+,,,3831074,programming, ,
+,,,1013428,programming, ,

Assignment-3/Week1/Validation_fold_csv/val_fold_4.csv

+URL ,Purpose,Solution,post_id,category,combined_text,processed_text
+,,,6628016,programming, ,
+,,,19297297,programming, ,
+,,,6484,programming, ,
+,,,10330998,programming, ,
+,,,10654608,programming, ,
+,,,5010528,programming, ,
+,,,10841635,programming, ,
+,,,3712017,programming, ,
+,,,4406656,programming, ,
+,,,1496497,programming, ,
+,,,720376,programming, ,
+,,,2121194,programming, ,
+,,,5107694,programming, ,
+,,,197482,programming, ,
+,,,5468906,programming, ,
+,,,16874549,programming, ,
+,,,10493559,programming, ,
+,,,4013779,programming, ,
+,,,172095,programming, ,
+,,,1330318,programming, ,
+,,,3081871,programming, ,
+,,,739268,programming, ,
+,,,4181668,programming, ,
+,,,2294010,programming, ,
+,,,8444267,programming, ,
+,,,1652941,programming, ,
+,,,9123457,programming, ,
+,,,5178757,programming, ,
+,,,3074700,programming, ,
+,,,6860053,programming, ,
+,,,6835358,programming, ,
+,,,546051,programming, ,
+,,,4069596,programming, ,
+,,,9207802,programming, ,
+,,,11434530,programming, ,
+,,,4461254,programming, ,
+,,,1478695,programming, ,
+,,,970828,programming, ,
+,,,426994,programming, ,
+,,,5904972,programming, ,
+,,,4544793,programming, ,
+,,,7714211,programming, ,
+,,,5606106,programming, ,
+,,,5366229,programming, ,
+,,,438669,programming, ,
+,,,13634364,programming, ,
+,,,7007929,programming, ,
+,,,600725,programming, ,
+,,,15088065,programming, ,
+,,,4787181,programming, ,
+,,,1369694,programming, ,
+,,,14518059,programming, ,
+,,,11102353,programming, ,
+,,,27312099,programming, ,
+,,,410391,programming, ,
+,,,5967894,programming, ,
+,,,6208998,programming, ,
+,,,1891119,programming, ,
+,,,2458631,programming, ,
+,,,5809963,programming, ,
+,,,2178897,programming, ,
+,,,1037925,programming, ,
+,,,643241,programming, ,
+,,,4913953,programming, ,
+,,,47683,programming, ,
+,,,4440957,programming, ,
+,,,12412163,programming, ,
+,,,22513658,programming, ,
+,,,4004279,programming, ,
+,,,657939,programming, ,
+,,,3119332,programming, ,
+,,,22094309,programming, ,
+,,,8150232,programming, ,
+,,,5552695,programming, ,
+,,,141781,programming, ,
+,,,3067735,programming, ,
+,,,5200024,programming, ,
+,,,18395523,programming, ,
+,,,3846107,programming, ,
+,,,2330381,programming, ,
+,,,8165398,programming, ,
+,,,4718262,programming, ,
+,,,11167793,programming, ,
+,,,5108145,programming, ,
+,,,6515018,programming, ,
+,,,18202999,programming, ,
+,,,774648,programming, ,
+,,,7005187,programming, ,
+,,,11076790,programming, ,
+,,,4272478,programming, ,
+,,,36999,programming, ,
+,,,819793,programming, ,
+,,,233571,programming, ,
+,,,3522544,programming, ,
+,,,3678121,programming, ,
+,,,364187,programming, ,
+,,,57701,programming, ,
+,,,1756541,programming, ,
+,,,9129750,programming, ,
+,,,4046,programming, ,
+,,,6648244,programming, ,
+,,,780880,programming, ,
+,,,4767102,programming, ,
+,,,20467904,programming, ,
+,,,4932722,programming, ,
+,,,15007281,programming, ,
+,,,8008672,programming, ,
+,,,13257114,programming, ,
+,,,11089368,programming, ,
+,,,16103492,programming, ,
+,,,11558237,programming, ,
+,,,21211848,programming, ,
+,,,27429099,programming, ,
+,,,3083186,programming, ,
+,,,14699195,programming, ,
+,,,656934,programming, ,
+,,,5897250,programming, ,
+,,,14730563,programming, ,
+,,,12041246,programming, ,
+,,,4792269,programming, ,
+,,,10676517,programming, ,
+,,,690177,programming, ,
+,,,11010151,programming, ,
+,,,2129820,programming, ,
+,,,2776731,programming, ,
+,,,2123430,programming, ,
+,,,4510903,programming, ,
+,,,10926621,programming, ,
+,,,4626642,programming, ,
+,,,262134,programming, ,
+,,,140104,programming, ,
+,,,1630540,programming, ,
+,,,14515084,programming, ,
+,,,13761340,programming, ,
+,,,10332158,programming, ,
+,,,2593628,programming, ,
+,,,2779757,programming, ,
+,,,718377,programming, ,
+,,,835839,programming, ,
+,,,1200346,programming, ,
+,,,8541485,programming, ,
+,,,7437818,programming, ,
+,,,1077747,programming, ,
+,,,8861183,programming, ,
+,,,4175428,programming, ,
+,,,7076576,programming, ,
+,,,7840088,programming, ,
+,,,6958396,programming, ,
+,,,15223779,programming, ,
+,,,5047196,programming, ,
+,,,20091836,programming, ,
+,,,10119544,programming, ,
+,,,11679651,programming, ,
+,,,6742938,programming, ,
+,,,6178815,programming, ,
+,,,3715778,programming, ,
+,,,2717596,programming, ,
+,,,7732371,programming, ,
+,,,14516191,programming, ,
+,,,859112,programming, ,
+,,,3274525,programming, ,
+,,,9599840,programming, ,
+,,,2777631,programming, ,
+,,,6788240,programming, ,
+,,,8317408,programming, ,
+,,,2792539,programming, ,
+,,,2842422,programming, ,
+,,,10150312,programming, ,
+,,,5510371,programming, ,
+,,,13224360,programming, ,
+,,,3831702,programming, ,
+,,,1217096,programming, ,
+,,,9187248,programming, ,
+,,,7748753,programming, ,
+,,,2327137,programming, ,
+,,,3611447,programming, ,
+,,,3302714,programming, ,
+,,,1065844,programming, ,
+,,,7897555,programming, ,
+,,,8511933,programming, ,
+,,,1510185,programming, ,
+,,,4603787,programming, ,
+,,,6065081,programming, ,
+,,,328485,programming, ,
+,,,8015840,programming, ,
+,,,9853453,programming, ,
+,,,424358,programming, ,
+,,,2011483,programming, ,
+,,,2379917,programming, ,
+,,,2727920,programming, ,
+,,,2587645,programming, ,
+,,,6123301,programming, ,
+,,,2280639,programming, ,
+,,,575021,programming, ,
+,,,2161674,programming, ,
+,,,4806314,programming, ,
+,,,511793,programming, ,

Assignment-3/Week1/load_preprocess_determine_terms.py

+import pandas as pd
+import re
+import nltk
+from nltk.stem import WordNetLemmatizer, PorterStemmer
+from bs4 import BeautifulSoup
+from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
+import os
+
+nltk.download('wordnet')
+nltk.download('punkt')
+
+
+# Step 1: Load and Combine Data
+def load_and_combine_data():
+    # Load data
+    architectural_df = pd.read_excel(
+        '/Users/abhayvaghasiya/Desktop/DS4SE_2/Assignment-3/Week-1/Architectural_Posts.xlsx')  
+    with open('/Users/abhayvaghasiya/Desktop/DS4SE_2/Assignment-3/Week-1/Programming_posts.rtf', 'r') as f:
+        programming_ids = f.read().strip().split(',')
+
+    # Create DataFrame for programming posts
+    programming_df = pd.DataFrame({'post_id': programming_ids, 'category': 'programming'})
+
+    # Combine both datasets
+    all_posts_df = pd.concat([architectural_df, programming_df], ignore_index=True)
+    all_posts_df.to_csv('combined_posts.csv', index=False)
+    print("Data loaded and combined successfully.")
+    print("Columns in the combined DataFrame:", all_posts_df.columns)
+    return all_posts_df
+
+
+# Step 2: Perform NLP Pre-processing
+def preprocess_text(text):
+    # Remove HTML tags
+    text = BeautifulSoup(text, "html.parser").get_text()
+
+    # Remove code snippets
+    text = re.sub(r'<code>(.*?)</code>', ' ', text)
+
+    # Tokenize text
+    tokens = nltk.word_tokenize(text)
+
+    # Remove stop words and non-alphabetic tokens
+    tokens = [word for word in tokens if word.isalpha() and word.lower() not in ENGLISH_STOP_WORDS]
+
+    # Lemmatize and stem
+    lemmatizer = WordNetLemmatizer()
+    stemmer = PorterStemmer()
+    tokens = [stemmer.stem(lemmatizer.lemmatize(word.lower())) for word in tokens]
+
+    return ' '.join(tokens)
+
+
+def perform_preprocessing(df):
+    # Use 'Purpose' and 'Solution' for concatenation instead of 'title', 'question', 'answer'
+    if all(col in df.columns for col in ['Purpose', 'Solution']):
+        df['combined_text'] = df[['Purpose', 'Solution']].fillna('').agg(' '.join, axis=1)
+    else:
+        print("Columns 'Purpose', 'Solution' are not present in the DataFrame.")
+        return df
+
+    df['processed_text'] = df['combined_text'].apply(preprocess_text)
+    df.to_csv('processed_posts.csv', index=False)
+    print("Text preprocessing completed successfully.")
+    print("Processed data saved to 'processed_posts.csv'.")
+    return df
+
+
+# Step 3: Determine Number of Terms per Post
+def determine_terms_per_post(df):
+    if 'processed_text' in df.columns:
+        df['term_count'] = df['processed_text'].apply(lambda x: len(x.split()))
+        average_terms_per_post = df['term_count'].mean()
+        print(f'Average number of terms per post: {average_terms_per_post}')
+    else:
+        print("'processed_text' column is not present in the DataFrame.")
+
+
+# Main function to run the combined steps
+if __name__ == "__main__":
+    all_posts_df = load_and_combine_data()
+    processed_posts_df = perform_preprocessing(all_posts_df)
+    determine_terms_per_post(processed_posts_df)
+    # Check if the processed_posts.csv file exists
+    if os.path.exists('processed_posts.csv'):
+        print("processed_posts.csv file is successfully created in the current directory.")
+    else:
+        print("Error: processed_posts.csv file was not created.")