Comments on: Sitemap Creator 0.2a: Create sitemaps 09 valid for Google, Yahoo, MSN, Ask.com and moreover sitemaps

By: wkarim

wkarim — Tue, 22 Jul 2008 01:47:53 +0000

@Frank
Good point! It has not been tested from console, however I will consider adding console interface on the next versions

By: Frank

Frank — Mon, 21 Jul 2008 17:23:02 +0000

Hi,
first, thank you for this great script. It works perfect with a browser. But it doesn't work from console. The only output is the HTML Markup from sitemap.php. But nothing is crawled and no sitemap is created. Do you have a workaround how to crawl from console/cron?
Thank You

By: bispak

bispak — Fri, 11 Jul 2008 05:01:24 +0000

when i download, install and run sitemap creator 0.2b
Voila.. it’s works well.. yippeeee….

sitemap20080711.xml.gz Created successfully

Where i can see sitemap20080711.xml.gz ?

when i'll submit sitemap to google as per SMC advised i dont know what i can submit since i cant find xml file under root or sitemap folder.

Please more advise and sorry to make you busy with my questions. Hope you dont mind.

By: bispak

bispak — Fri, 11 Jul 2008 04:20:41 +0000

thanks, i'll download and try sitemap creator 0.2b

By: wkarim

wkarim — Wed, 09 Jul 2008 19:46:12 +0000

@bispak
please use the new version sitemap creator 0.2b

By: bispak

bispak — Wed, 09 Jul 2008 00:58:06 +0000

same result happened with or without “www".
success in crawl but csv file and sitemap not created.
what’s wrong with me? am i not lucky?

By: bispak

bispak — Wed, 09 Jul 2008 00:11:24 +0000

When i clink on “Display CSV File" the result is same with when i click on “Create Sitemaps".
“Please crawl tukarinfobispak.com first"

By: bispak

bispak — Wed, 09 Jul 2008 00:07:24 +0000

i'm sure crawling is finish with the result:
Finished crawling http://www.tukarinfobispak.com, Crawled 72 links
Took 26.69 Seconds, using 2MB of memory

and then at the end mention about the url to add to crontab or schedule tasks.

Still, when i click “Create Sitemaps" i got answer: “Please crawl tukarinfobispak.com first".

I use PHP Version 5.2.5

By: wkarim

wkarim — Tue, 08 Jul 2008 14:53:59 +0000

@bispak
Make sure the script finishes crawling before you create the sitemap

By: gtnman

gtnman — Fri, 09 May 2008 19:34:30 +0000

I turned off safe mode, however when I crawl the site it crawls perfect then I click create sitemap, and it tell me I must crawl the site first.

I have curl disabled and www set to true.

By: gtnman

gtnman — Wed, 07 May 2008 21:22:44 +0000

I assume that this script does not work w/ PHP safemode?

By: cyberswept

cyberswept — Thu, 01 May 2008 22:22:04 +0000

sl I changed the .function.inc.php as you described and it now crawls my site, but when I go to create the sitemaps it says, please crawl domain.com first. Any suggestions?

By: Gsm files center

Gsm files center — Sat, 26 Apr 2008 23:10:36 +0000

Thanks for the sitemap script, it’s working great
nice work ..

By: sl

sl — Wed, 23 Apr 2008 17:00:40 +0000

stripos() and curl_setopt_array() is not php 4.x compilant. so i fixed following lines in file .function.inc.php.

line 217:
$ch = curl_init($url);

changed to:
foreach ($options as $myoption => $myvalue) {
curl_setopt ($ch, $myoption, $myvalue);
}
–
line 233:
if ( isset($header['content_type']) && stripos($header['content_type'], 'text') === false ){

changed to:
if ( isset($header['content_type']) && strpos(strtolower($header['content_type']), 'text') === false ){
–

By: wkarim

wkarim — Mon, 14 Apr 2008 12:45:23 +0000

@shawn
- No, you mostly do not need to edit this.
- For crawling more than one domain you need to keep SMC_SITE as $_SERVER['HTTP_HOST'] and run the script from these different domains, every domain will create its own sitemap and will product a different sitemap which you can submit to Google webmaster’s CP.
- You are welcome to show a link back to Sitemap Creator.

By: shawn

shawn — Mon, 14 Apr 2008 12:42:19 +0000

sorry i do have this sitemap.php file and it is in the root directory.and there is also referance in the robots.txt to it!

i have 3 domains and want them all crawled seperate and they seem to be working but what do i need to tell google my site map is? sitemap.php?

if we get this working properly i will help you promote it my placing a link on my sites!

By: shawn

shawn — Mon, 14 Apr 2008 12:20:01 +0000

so you do need to add any thing to these
Google' => 'http://www.google.com/webmasters/sitemaps/ping?sitemap=',
'Yahoo' => 'http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=SitemapWriter&url=',
'Live Search' => 'http://webmaster.live.com/ping.aspx?siteMap=',
'Ask.com' => 'http://submissions.ask.com/ping?sitemap=',
'MoreOver' => 'http://api.moreover.com/ping?u=',?

By: wkarim

wkarim — Sat, 12 Apr 2008 18:05:14 +0000

@shawn
- you do not need to edit internal settings.
- the script provides another page ’sitemap.php' which should be placed in the root directory where it redirects the bot to the sitemap location.
- There is an option on the admin area to add the sitemap URL to robots.txt
- The script creates a new sitemap if the time of crawling is one day different from the last created sitemap, you can create a cron job to delete old sitemaps
- The sitemap on the posted URL does not exist, please regenerate it.

By: shawn

shawn — Tue, 08 Apr 2008 17:26:39 +0000

Sitemap : http://floafieds.com/sitemap.php?do=showsitemap&sm=sitemap.xml.gz
when google runs this they get a server internal error is there any fix?

By: shawn

shawn — Tue, 08 Apr 2008 12:29:33 +0000

wkarim,

can you tell me if the
/*internal settings*/
define('SMC_VERSION', '0.2a');
define('SMC_DATA_CACHE', SMC_DATA.'cache/');
define('SMC_DATA_SITES', SMC_DATA.’sites/');
define('SMC_DATA_ERRORS', SMC_DATA.'errors/');
define('SMC_DATA_SITEMAPS', SMC_DATA_SITES.SMC_SITE.'_sitemaps/');
$pings = array (
'Google' => 'http://www.google.com/webmasters/sitemaps/ping?sitemap=',
'Yahoo' => 'http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=SitemapWriter&url=',
'Live Search' => 'http://webmaster.live.com/ping.aspx?siteMap=',
'Ask.com' => 'http://submissions.ask.com/ping?sitemap=',
'MoreOver' => 'http://api.moreover.com/ping?u=',
);
does any of this have to be edited and will this script automatacly ping the search engines
because google askes for sitemap.xml in your root directory and the script does not save one there that i can see please explain how this works!! the scrips is awsome also in the admin section it would be nice if the admin could edit or add to the robots.txt file i have set up a cron task to run every hour it seems to work but does it save a new site map in the cron task or just update the existing one i have an auction site so it will be changing all the time.

By: wkarim

wkarim — Thu, 27 Mar 2008 07:04:32 +0000

@shawn
glad it helped, thanks for nice words

By: shawn

shawn — Thu, 27 Mar 2008 06:25:15 +0000

this is just what i was looking for for a week and 50 scripts later yep works like a charm my site is huge so i uped the mem to 600 and time out to 1800 and it rolls like a swiss watch

I CAN NOT SAY THAK YOU ENOUGH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

By: Sitemap Creator - Create sitemaps for Google, Yahoo, MSN and Ask.comphp and javascript

Sun, 23 Mar 2008 20:18:38 +0000

[...] URL: http://gadelkareem.com/2007/12/10/sitemap-creator-02a-create-sitemaps-09-valid-for-google-yahoo-and-... [...]

By: wkarim

wkarim — Fri, 29 Feb 2008 23:34:26 +0000

@Tammy
please change SMC_USE_WWW to true and add www to your domain as all links displayed there include www

By: Tammy

Tammy — Fri, 29 Feb 2008 16:49:08 +0000

thank you wkarim, I do that but now when it crawls it goes through the whole 350 seconds and then tells me:
No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host

By: wkarim

wkarim — Wed, 27 Feb 2008 20:21:39 +0000

@Tammy
This might be related to concurrent connection or bandwidth limit per session on your server.
To solve this problem try commenting line :
$get .= “Connection: close\r\n\r\n";
on function.inc.php file

By: Tammy

Tammy — Wed, 27 Feb 2008 19:40:04 +0000

I keep getting this error when crawling. I've tried raising the timeout to no avail

WARNING: Connection failed (0)
WARNING: Connection failed (0)
NOTICE: Document type is application/xml for URL http://mysite.com/clients/announcements.xml
WARNING: Connection failed (0)
NOTICE: Document type is application/zip for URL http://mysite.com/themes/dreamland.zip
NOTICE: Document type is application/zip for URL http://mysite.com/themes/Green-Glow_000.zip
NOTICE: Document type is application/zip for URL http://mysite.com/themes/mountain-dawn.zip
NOTICE: Document type is application/zip for URL http://mysite.com/themes/yalla.zip

By: wkarim

wkarim — Tue, 26 Feb 2008 03:35:25 +0000

@Darren
I am afraid this is not available, I will try to include that as an option on the config file on next releases.
A workaround is to edit the csv file and delete links you do not want to include on the sitemap. Or in case you do not want to crawl that directory you may blacklist it.

By: Darren

Darren — Tue, 26 Feb 2008 03:18:04 +0000

Is there a way to limit the depth in a single directory. I have a calendar and once it goes into that area it fills it up fast.

By: wkarim

wkarim — Fri, 22 Feb 2008 02:25:16 +0000

@Andrew
Thank you for your comment.

By: Andrew

Andrew — Thu, 21 Feb 2008 19:57:29 +0000

Guys and Girls..I downloaded the most recent version on hotscripts.com and edited config.php.inc. All I changed was the prefix for my email and the password (did not touch the HTTP_HOST). I uploaded the files to my site, chmod 777 the data folder, and accessed the site via this link: http://domain.com/sitemap/sitemap.php (NO WWW)

It went through and indexed everything fine. Works like a charm, thanks webmaster!

By: wkarim

wkarim — Mon, 18 Feb 2008 17:46:53 +0000

@Dhyar IRdiansyah
make sure the links on the page have the same domain as your localhost or make changes on the config file

By: Dhyar IRdiansyah

Dhyar IRdiansyah — Mon, 18 Feb 2008 17:42:41 +0000

I get error “No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host". what must i do?

By: wkarim

wkarim — Mon, 18 Feb 2008 15:17:05 +0000

@neeraj
as far as there’s a link to the forums it should crawl it with no problem

By: neeraj

neeraj — Mon, 18 Feb 2008 11:25:20 +0000

hello
i have install it at mmmec.com but i have a forum under mmmec.com/forum ,so how can i make it to map my forum also ,where should i put the link depth for it ??

By: wkarim

wkarim — Sat, 16 Feb 2008 14:05:25 +0000

@Ruud
yes, That should work

By: Ruud

Ruud — Sat, 16 Feb 2008 13:56:19 +0000

this way I access the script:

http://bestplaces2be.com/sitemap/sitemap.php

By: wkarim

wkarim — Sat, 16 Feb 2008 12:48:46 +0000

@Ruud
if you're using $_SERVER[’HTTP_HOST’] then you need to access the script from http://example.com not http://www.example.com

By: Ruud

Ruud — Sat, 16 Feb 2008 12:35:13 +0000

I use your last build
Sitemap Creator 0.2 alpha build 20080109

and use this in config.inc.php
define('SMC_SITE', $_SERVER['HTTP_HOST']);

if ( isset($header['content_type']) && strpos(strtolower($header[’content_type’]), 'text') === false ){
_error("Document type is {$header['content_type']} for URL {$url}“);

after this it stops I think

By: wkarim

wkarim — Thu, 14 Feb 2008 16:19:55 +0000

@webmaster
For the warnings you weather need to fix permissions or disable the use for curl from the config file.

1. Yes, there are two sitemap.php files, and you are supposed to use the one inside the sitemap folder
to extract directly on your server do
tar xzf sitemap_creator.tar.gz

2. same as previous

By: wkarim

wkarim — Thu, 14 Feb 2008 16:09:55 +0000

@Ruud
Remove all edited php files and replace them with the ones from the recent build.
Do not add 'http://' at the beginning of your domain

By: webmaster

webmaster — Thu, 14 Feb 2008 01:24:49 +0000

Hi,

Now I get:

Warning: tempnam() [function.tempnam]: open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/var/www/virtual/anysite.com/:/usr/share/php/:/tmp/) in /var/www/virtual/anysite.com/htdocs/sitemap/.function.inc.php on line 87

Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(/smc_cookies) is not within the allowed path(s): (/var/www/virtual/anysite.com/:/usr/share/php/:/tmp/) in /var/www/virtual/anysite.com/htdocs/sitemap/.function.inc.php on line 143

Also, I have two questions:

1. I see a folder with sitemaps. Is there supposed to be a seperate sitemaps.php for the root and then a seperate folder called sitemap.

2. The script keeps stating that my “data" folder may not exist or might not be writable. However, it does exist and is writable.

By: Ruud

Ruud — Wed, 13 Feb 2008 12:17:32 +0000

please replace the files with the original ones
What do you mean with this?
I changed everywhere the Http://www.bestplaces2be.com into http://bestpalces2be.com

cache and errors empty, result the same

By: wkarim

wkarim — Tue, 12 Feb 2008 13:01:03 +0000

@wbmaster
that’s not the same line on the last build, please re download.

By: webmaster

webmaster — Tue, 12 Feb 2008 12:42:09 +0000

script language="javascript" type="text/javascript

I had to remove the brackets to post.

By: wkarim

wkarim — Mon, 11 Feb 2008 16:53:11 +0000

@webmaster
can you copy and paste line 54 on sitemap.php

By: webmaster

webmaster — Mon, 11 Feb 2008 15:38:49 +0000

Hi,

I have downloaded the most recent script and still get the same error.

By: wkarim

wkarim — Mon, 11 Feb 2008 14:56:40 +0000

@Ruud
please replace the files with the original ones and empty cache and error folders.
That’s not a possible error!

By: Ruud

Ruud — Mon, 11 Feb 2008 11:45:48 +0000

okay removed the www now still get this:
NOTICE: Document type is text/html for URL http://bestplaces2be.com/md/index.php
NOTICE: Document type is text/html for URL http://bestplaces2be.com/
No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host

By: wkarim

wkarim — Sat, 09 Feb 2008 19:03:45 +0000

@webmaster
please download the new build, it should fix this problem

By: webmaster

webmaster — Sat, 09 Feb 2008 16:39:51 +0000

Warning: Cannot modify header information - headers already sent by (output started at /var/www/virtual/luvshades.com/htdocs/sitemap.php:54) in /var/www/virtual/luvshades.com/htdocs/.function.inc.php on line 703

When I try any function, it goes back to sign-in page.

By: wkarim

wkarim — Tue, 05 Feb 2008 21:59:51 +0000

@Ruud
you have your domain set with www at the beginning while no link on the pages begins with 'www' The script is able to crawl up-level sub-domains but not the reverse.
Set your domain without 'www'

By: Ruud

Ruud — Tue, 05 Feb 2008 17:36:34 +0000

some additional info:
CURL :
define(’SMC_USE_CURL’, false);
Then clean your cache and blacklist folders as the index page might has been blacklisted

I did this also.

By: Ruud

Ruud — Tue, 05 Feb 2008 17:34:59 +0000

Hi,
Get the following message:

No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host

I've index.php in the root folder which redirects to http://www.bestplaces2be.com/md/index.php

index.php

any idee what goes wrong here?

By: wkarim

wkarim — Fri, 01 Feb 2008 22:20:21 +0000

@pfff
I think you do not have enough privileges to run curl on your server, please disable CURL :
define('SMC_USE_CURL', false);
Then clean your cache and blacklist folders as the index page might has been blacklisted

By: Pfff

Pfff — Wed, 30 Jan 2008 19:46:41 +0000

Hello,

i just solve the matter by uploading the config file without any change BUT now, my website can't be crawled …

here’s the error message:

No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host

but idon't know how to set the domain …

Please help

By: Pfff

Pfff — Wed, 30 Jan 2008 06:29:17 +0000

Warning: tempnam() [function.tempnam]: open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/www/tmp/:/usr/local/lib/php:/usr/local/bin:/home/www/pfff) in /home/www/pfff/www/sitemap/.function.inc.php on line 87

Warning: curl_setopt_array() [function.curl-setopt-array]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in /home/www/pfff/www/sitemap/.function.inc.php on line 218
ERROR6: Couldn't resolve host 'wwwwebynuxnet' for URL http://wwwwebynuxnet/

No Pages were crawled, Please make sure you have set your site domain correctly and you have valid connection to host

why the “." are note between www, webynux and net ?

By: wkarim

wkarim — Tue, 29 Jan 2008 20:05:06 +0000

you need to access the sitemap through
/sitemap/sitemap.php
I think you have disabled error reporting from your php.ini file, you need to enable that to check what error you have.
also, please let me know your PHP version

By: Pfff

Pfff — Tue, 29 Jan 2008 19:36:15 +0000

Hello,

i need some help to install this script.
http://www.webynux.net/sitemap.php is a blank page …

i think i've problem to configure the .config.inc.php.

Could you please help me ??

By: wkarim

wkarim — Sun, 27 Jan 2008 22:26:12 +0000

@Mafiozy
yes it’s working

By: Mafiozy

Mafiozy — Thu, 24 Jan 2008 13:19:38 +0000

Does this thing works???

By: wkarim

wkarim — Fri, 18 Jan 2008 16:59:19 +0000

@shaun
yeah sorry for that one…you can use this instead
strpos(strtolower($header['content_type']), 'text')
I guess the script might need PHP5 to work, anyway let me know if it worked with this fix

By: shaun

shaun — Fri, 18 Jan 2008 16:52:48 +0000

ahh I'm running php 4.4.8. stripos is for php 5. I changed stripos to strpos and now it is running how ever it isn't going to be case insensitive now. Not sure what the result will be but it is running.

By: shaun

shaun — Fri, 18 Jan 2008 16:40:53 +0000

I click crawl site. and it says crawling please wait and after 4 seconds I get this error.

Fatal error: Call to undefined function: stripos() in /home/shaun/public_html/site/sitemap/.function.inc.php on line 233

By: wkarim

wkarim — Fri, 18 Jan 2008 05:50:29 +0000

@Reiner
Current version of Sitemap Creator 0.2a does not have exclude directory feature, you might edit the created CSV file with any text editor to do that.
blacklist directory path depends on your configuration, default is sitemap/data/errors/ or you can change it through config file by changing SMC_DATA_ERRORS

By: Reiner

Reiner — Wed, 16 Jan 2008 14:57:56 +0000

Hi,
How do I exclude the directroies I don't wan't to be listed, crawled BEFORE running the script and where is the blacklist file or directory?

By: wkarim

wkarim — Sat, 12 Jan 2008 16:23:00 +0000

Please re download and extract the files and run without any editing, it appears that you've added unneeded characters in the config file

By: Paddy

Paddy — Sat, 12 Jan 2008 08:51:58 +0000

i don't know what i have done but i get lots of errors:

Warning: Division by zero in C:\xampp\htdocs\New Folder\sitemap\.config.inc.php on line 52

Warning: Cannot modify header information - headers already sent by (output started at C:\xampp\htdocs\New Folder\sitemap\.config.inc.php:52) in C:\xampp\htdocs\New Folder\sitemap\sitemap.php on line 49

Warning: file(C:\xampp\htdocs\New Folder\sitemap/data/sites/) [function.file]: failed to open stream: No such file or directory in C:\xampp\htdocs\New Folder\sitemap\.function.inc.php on line 570

Warning: Invalid argument supplied for foreach() in C:\xampp\htdocs\New Folder\sitemap\.function.inc.php on line 572

i dont know whats happining can anyone help me

By: nit

nit — Wed, 09 Jan 2008 13:16:58 +0000

np thx for this nice script..

By: wkarim

wkarim — Wed, 09 Jan 2008 11:58:00 +0000

@nit
Thank you for reporting this error, I'm not sure how I didn't get that on my server!
a new build is available with all reported errors fixed

By: nit

nit — Wed, 09 Jan 2008 01:58:09 +0000

forgot to add;

the redirect doesnt work too. it loads for a moment then its a blank page..

every link puts me back to the login screen

By: nit

nit — Wed, 09 Jan 2008 01:55:41 +0000

after i insall and edit the config file and go to web and enter the password i put in the config file I get this;

Warning: Cannot modify header information - headers already sent by (output started at /home/osgames/public_html/sitemap/sitemap/sitemap.php:54) in /home/osgames/public_html/sitemap/sitemap/.function.inc.php on line 703

when i click on the crawl site link it puts me back to the password screen and it just goes on and on..

php -v
PHP 5.2.1 (cli) (built: Feb 23 2007 08:00:24)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
with Zend Extension Manager v1.2.0, Copyright (c) 2003-2006, by Zend Technologies
with Zend Optimizer v3.2.2, Copyright (c) 1998-2006, by Zend Technologies

mysql is 4.1.22

thanks..

By: wkarim

wkarim — Sun, 06 Jan 2008 23:51:26 +0000

The script is tested against PHP v5.2.5 However, I tried to make it as compatible as possible with previous versions of PHP the error 'Parse error: syntax error, unexpected ‘=’, expecting ‘)’ in /.function.inc.php on line 99' is due to using a default value on a variable passed by reference which is not supported on PHP 4.2.2 Solution : ~~download Sitemap Creator 0.2 for 4.2.2~~ fixed on the new build

By: wkarim

wkarim — Sun, 06 Jan 2008 23:20:37 +0000

@Chris
you either should recompile PHP with curl library support, or change 'SMC_USE_CURL' on the configuration file to false.

By: Chris

Chris — Sun, 06 Jan 2008 15:33:31 +0000

Had the same error as Jerry, removed the “&" in various lines with errors.
Now the program starts but if I click on “Crawl" I get the following error: “Call to undefined function: curl_setopt_array() in /www/htdocs/v031207/.function.inc.php on line 218″
My php-version: 4.4.8

By: wkarim

wkarim — Sun, 06 Jan 2008 03:42:57 +0000

@Flash Buddy:
what PHP version are you using?
a fast answer would be to remove the '&' before the '$val' on that line.
Let me know if it worked

By: Flash Buddy

Flash Buddy — Thu, 03 Jan 2008 23:54:50 +0000

I get:
Parse error: syntax error, unexpected '=', expecting ')' in /home/hqcodec/public_html/.function.inc.php on line 99

By: Mike

Mike — Fri, 21 Dec 2007 16:45:09 +0000

I've also tried with no success (although I'm not proficient with PHP). Are there more detailed instructions available?

By: wkarim

wkarim — Fri, 21 Dec 2007 08:50:31 +0000

@Jerry : Can you describe the error?

By: Jerry

Jerry — Wed, 19 Dec 2007 00:06:54 +0000

Has anyone got this to work? Have spent several hours on it. No luck, can not get it to redirect, I think.

By: Sitemap Creator 0.1 : Create Sitemaps 0.9 valid for Google, Yahoo! and MSN Sitemaps :: GadElKareem

Mon, 10 Dec 2007 09:34:01 +0000

[...] New Sitemap Creator 0.2a available [...]